Next: , Previous: , Up: asdf: another system definition facility   [Contents][Index]


12 FAQ

12.1 “Where do I report a bug?”

ASDF bugs are tracked on launchpad: https://launchpad.net/asdf.

If you’re unsure about whether something is a bug, or for general discussion, use the asdf-devel mailing list

12.2 “What has changed between ASDF 1 and ASDF 2?”

12.2.1 What are ASDF 1 and ASDF 2?

On May 31st 2010, we have released ASDF 2. ASDF 2 refers to release 2.000 and later. (Releases between 1.656 and 1.728 were development releases for ASDF 2.) ASDF 1 to any release earlier than 1.369 or so. If your ASDF doesn’t sport a version, it’s an old ASDF 1.

ASDF 2 and its release candidates push :asdf2 onto *features* so that if you are writing ASDF-dependent code you may check for this feature to see if the new API is present. All versions of ASDF should have the :asdf feature.

If you are experiencing problems or limitations of any sort with ASDF 1, we recommend that you should upgrade to ASDF 2, or whatever is the latest release.

12.2.2 ASDF can portably name files in subdirectories

Common Lisp namestrings are not portable, except maybe for logical pathnamestrings, that themselves have various limitations and require a lot of setup that is itself ultimately non-portable.

In ASDF 1, the only portable ways to refer to pathnames inside systems and components were very awkward, using #.(make-pathname ...) and #.(merge-pathnames ...). Even the above were themselves were inadequate in the general case due to host and device issues, unless horribly complex patterns were used. Plenty of simple cases that looked portable actually weren’t, leading to much confusion and greavance.

ASDF 2 implements its own portable syntax for strings as pathname specifiers. Naming files within a system definition becomes easy and portable again. See asdf:system-relative-pathname, asdf-utilities:merge-pathnames*, asdf::merge-component-name-type.

On the other hand, there are places where systems used to accept namestrings where you must now use an explicit pathname object: (defsystem ... :pathname "LOGICAL-HOST:PATH;TO;SYSTEM;" ...) must now be written with the #p syntax: (defsystem ... :pathname #p"LOGICAL-HOST:PATH;TO;SYSTEM;" ...)

See Pathname specifiers.

12.2.3 Output translations

A popular feature added to ASDF was output pathname translation: asdf-binary-locations, common-lisp-controller, cl-launch and other hacks were all implementing it in ways both mutually incompatible and difficult to configure.

Output pathname translation is essential to share source directories of portable systems across multiple implementations or variants thereof, or source directories of shared installations of systems across multiple users, or combinations of the above.

In ASDF 2, a standard mechanism is provided for that, asdf-output-translations, with sensible defaults, adequate configuration languages, a coherent set of configuration files and hooks, and support for non-Unix platforms.

See Controlling where ASDF saves compiled files.

12.2.4 Source Registry Configuration

Configuring ASDF used to require special magic to be applied just at the right moment, between the moment ASDF is loaded and the moment it is used, in a way that is specific to the user, the implementation he is using and the application he is building.

This made for awkward configuration files and startup scripts that could not be shared between users, managed by administrators or packaged by distributions.

ASDF 2 provides a well-documented way to configure ASDF, with sensible defaults, adequate configuration languages, and a coherent set of configuration files and hooks.

We believe it’s a vast improvement because it decouples application distribution from library distribution. The application writer can avoid thinking where the libraries are, and the library distributor (dpkg, clbuild, advanced user, etc.) can configure them once and for every application. Yet settings can be easily overridden where needed, so whoever needs control has exactly as much as required.

At the same time, ASDF 2 remains compatible with the old magic you may have in your build scripts (using *central-registry* and *system-definition-search-functions*) to tailor the ASDF configuration to your build automation needs, and also allows for new magic, simpler and more powerful magic.

See Controlling where ASDF searches for systems.

12.2.5 Usual operations are made easier to the user

In ASDF 1, you had to use the awkward syntax (asdf:oos 'asdf:load-op :foo) to load a system, and similarly for compile-op, test-op.

In ASDF 2, you can use shortcuts for the usual operations: (asdf:load-system :foo), and similarly for compile-system, test-system.

12.2.6 Many bugs have been fixed

The following issues and many others have been fixed:

  • The infamous TRAVERSE function has been revamped significantly, with many bugs squashed. In particular, dependencies were not correctly propagated across submodules within a system but now are. The :version and :feature features and the :force (system1 .. systemN) feature have been fixed.
  • Performance has been notably improved for large systems (say with thousands of components) by using hash-tables instead of linear search, and linear-time list accumulation instead of quadratic-time recursive appends.
  • Many features used to not be portable, especially where pathnames were involved. Windows support was notably quirky because of such non-portability.
  • The internal test suite used to massively fail on many implementations. While still incomplete, it now fully passes on all implementations supported by the test suite, except for GCL (due to GCL bugs).
  • Support was lacking for some implementations. ABCL and GCL were notably wholly broken. ECL extensions were not integrated in the ASDF release.
  • The documentation was grossly out of date.

12.2.7 ASDF itself is versioned

Between new features, old bugs fixed, and new bugs introduced, there were various releases of ASDF in the wild, and no simple way to check which release had which feature set. People using or writing systems had to either make worst-case assumptions as to what features were available and worked, or take great pains to have the correct version of ASDF installed.

With ASDF 2, we provide a new stable set of working features that everyone can rely on from now on. Use #+asdf2 to detect presence of ASDF 2, (asdf:version-satisfies (asdf:asdf-version) "2.000") to check the availability of a version no earlier than required.

12.2.8 ASDF can be upgraded

When an old version of ASDF was loaded, it was very hard to upgrade ASDF in your current image without breaking everything. Instead you have to exit the Lisp process and somehow arrange to start a new one from a simpler image. Something that can’t be done from within Lisp, making automation of it difficult, which compounded with difficulty in configuration, made the task quite hard. Yet as we saw before, the task would have been required to not have to live with the worst case or non-portable subset of ASDF features.

With ASDF 2, it is easy to upgrade from ASDF 2 to later versions from within Lisp, and not too hard to upgrade from ASDF 1 to ASDF 2 from within Lisp. We support hot upgrade of ASDF and any breakage is a bug that we will do our best to fix. There are still limitations on upgrade, though, most notably the fact that after you upgrade ASDF, you must also reload or upgrade all ASDF extensions.

12.2.9 Decoupled release cycle

When vendors were releasing their Lisp implementations with ASDF, they had to basically never change version because neither upgrade nor downgrade was possible without breaking something for someone, and no obvious upgrade path was visible and recommendable.

With ASDF 2, upgrade is possible, easy and can be recommended. This means that vendors can safely ship a recent version of ASDF, confident that if a user isn’t fully satisfied, he can easily upgrade ASDF and deal with a supported recent version of it. This means that release cycles will be causally decoupled, the practical consequence of which will mean faster convergence towards the latest version for everyone.

12.2.10 Pitfalls of the transition to ASDF 2

The main pitfalls in upgrading to ASDF 2 seem to be related to the output translation mechanism.

  • Output translations is enabled by default. This may surprise some users, most of them in pleasant way (we hope), a few of them in an unpleasant way. It is trivial to disable output translations. See “How can I wholly disable the compiler output cache?”.
  • Some systems in the large have been known not to play well with output translations. They were relatively easy to fix. Once again, it is also easy to disable output translations, or to override its configuration.
  • The new ASDF output translations are incompatible with ASDF-Binary-Locations. They replace A-B-L, and there is compatibility mode to emulate your previous A-B-L configuration. See asdf:enable-asdf-binary-locations-compatibility in see Backward Compatibility. But thou shall not load ABL on top of ASDF 2.

Other issues include the following:

  • ASDF pathname designators are now specified in places where they were unspecified, and a few small adjustments have to be made to some non-portable defsystems. Notably, in the :pathname argument to a defsystem and its components, a logical pathname (or implementation-dependent hierarchical pathname) must now be specified with #p syntax where the namestring might have previously sufficed; moreover when evaluation is desired #. must be used, where it wasn’t necessary in the toplevel :pathname argument.
  • There is a slight performance bug, notably on SBCL, when initially searching for asd files, the implicit (directory "/configured/path/**/*.asd") for every configured path (:tree "/configured/path/") in your source-registry configuration can cause a slight pause. Try to (time (asdf:initialize-source-registry)) to see how bad it is or isn’t on your system. If you insist on not having this pause, you can avoid the pause by overriding the default source-registry configuration and not use any deep :tree entry but only :directory entries or shallow :tree entries. Or you can fix your implementation to not be quite that slow when recursing through directories.
  • On Windows, only LispWorks supports proper default configuration pathnames based on the Windows registry. Other implementations make do with environment variables. Windows support is somewhat less tested than Unix support. Please help report and fix bugs.
  • The mechanism by which one customizes a system so that Lisp files may use a different extension from the default .lisp has changed. Previously, the pathname for a component was lazily computed when operating on a system, and you would (defmethod source-file-type ((component cl-source-file) (system (eql (find-system 'foo)))) (declare (ignorable component system)) "cl"). Now, the pathname for a component is eagerly computed when defining the system, and instead you will (defclass my-cl-source-file (cl-source-file) ((type :iniform "cl"))) and use :default-component-class my-cl-source-file as argument to defsystem, as detailed in a see How do I create a system definition where all the source files have a .cl extension? below.

12.3 Issues with installing the proper version of ASDF

12.3.1 “My Common Lisp implementation comes with an outdated version of ASDF. What to do?”

We recommend you upgrade ASDF. See Upgrading ASDF.

If this does not work, it is a bug, and you should report it. See Where do I report a bug. In the meantime, you can load asdf.lisp directly. See Loading an otherwise installed ASDF.

12.3.2 “I’m a Common Lisp implementation vendor. When and how should I upgrade ASDF?”

Starting with current candidate releases of ASDF 2, it should always be a good time to upgrade to a recent ASDF. You may consult with the maintainer for which specific version they recommend, but the latest RELEASE should be correct. We trust you to thoroughly test it with your implementation before you release it. If there are any issues with the current release, it’s a bug that you should report upstream and that we will fix ASAP.

As to how to include ASDF, we recommend the following:

  • If ASDF isn’t loaded yet, then (require :asdf) should load the version of ASDF that is bundled with your system. You may have it load some other version configured by the user, if you allow such configuration.
  • If your system provides a mechanism to hook into CL:REQUIRE, then it would be nice to add ASDF to this hook the same way that ABCL, CCL, CLISP, CMUCL, ECL, SBCL and SCL do it.
  • You may, like SBCL, have ASDF be implicitly used to require systems that are bundled with your Lisp distribution. If you do have a few magic systems that come with your implementation in a precompiled way such that one should only use the binary version that goes with your distribution, like SBCL does, then you should add them in the beginning of wrapping-source-registry.
  • If you have magic systems as above, like SBCL does, then we explicitly ask you to NOT distribute asdf.asd as part of those magic systems. You should still include the file asdf.lisp in your source distribution and precompile it in your binary distribution, but asdf.asd if included at all, should be secluded from the magic systems, in a separate file hierarchy; alternatively, you may provide the system after renaming it and its .asd file to e.g. asdf-ecl and asdf-ecl.asd, or sb-asdf and sb-asdf.asd. Indeed, if you made asdf.asd a magic system, then users would no longer be able to upgrade ASDF using ASDF itself to some version of their preference that they maintain independently from your Lisp distribution.
  • If you do not have any such magic systems, or have other non-magic systems that you want to bundle with your implementation, then you may add them to the default-source-registry, and you are welcome to include asdf.asd amongst them.
  • Please send us upstream any patches you make to ASDF itself, so we can merge them back in for the benefit of your users when they upgrade to the upstream version.

12.4 Issues with configuring ASDF

12.4.1 “How can I customize where fasl files are stored?”

See Controlling where ASDF saves compiled files.

Note that in the past there was an add-on to ASDF called ASDF-binary-locations, developed by Gary King. That add-on has been merged into ASDF proper, then superseded by the asdf-output-translations facility.

Note that use of asdf-output-translations can interfere with one aspect of your systems — if your system uses *load-truename* to find files (e.g., if you have some data files stored with your program), then the relocation that this ASDF customization performs is likely to interfere. Use asdf:system-relative-pathname to locate a file in the source directory of some system, and use asdf:apply-output-translations to locate a file whose pathname has been translated by the facility.

12.4.2 “How can I wholly disable the compiler output cache?”

To permanently disable the compiler output cache for all future runs of ASDF, you can:

mkdir -p ~/.config/common-lisp/asdf-output-translations.conf.d/
echo ':disable-cache' > ~/.config/common-lisp/asdf-output-translations.conf.d/99-disable-cache.conf

This assumes that you didn’t otherwise configure the ASDF files (if you did, edit them again), and don’t somehow override the configuration at runtime with a shell variable (see below) or some other runtime command (e.g. some call to asdf:initialize-output-translations).

To disable the compiler output cache in Lisp processes run by your current shell, try (assuming bash or zsh) (on Unix and cygwin only):

export ASDF_OUTPUT_TRANSLATIONS=/:

To disable the compiler output cache just in the current Lisp process, use (after loading ASDF but before using it):

(asdf:disable-output-translations)

12.5 Issues with using and extending ASDF to define systems

12.5.1 “How can I cater for unit-testing in my system?”

ASDF provides a predefined test operation, test-op. See test-op. The test operation, however, is largely left to the system definer to specify. test-op has been a topic of considerable discussion on the asdf-devel mailing list, and on the launchpad bug-tracker.

Here are some guidelines:

  • For a given system, foo, you will want to define a corresponding test system, such as foo-test. The reason that you will want this separate system is that ASDF does not out of the box supply components that are conditionally loaded. So if you want to have source files (with the test definitions) that will not be loaded except when testing, they should be put elsewhere.
  • The foo-test system can be defined in an asd file of its own or together with foo. An aesthetic preference against cluttering up the filesystem with extra asd files should be balanced against the question of whether one might want to directly load foo-test. Typically one would not want to do this except in early stages of debugging.
  • Record that testing is implemented by foo-test. For example:
    (defsystem foo
       :in-order-to ((test-op (test-op foo-test)))
       ....)
    
    (defsystem foo-test
       :depends-on (foo my-test-library ...)
       ....)
    

This procedure will allow you to support users who do not wish to install your test framework.

One oddity of ASDF is that operate (see operate) does not return a value. So in current versions of ASDF there is no reliable programmatic means of determining whether or not a set of tests has passed, or which tests have failed. The user must simply read the console output. This limitation has been the subject of much discussion.

12.5.2 “How can I cater for documentation generation in my system?”

The ASDF developers are currently working to add a doc-op to the set of predefined ASDF operations. See Predefined operations of ASDF. See also https://bugs.launchpad.net/asdf/+bug/479470.

12.5.3 “How can I maintain non-Lisp (e.g. C) source files?”

See cffi’s cffi-grovel.

12.5.4 “I want to put my module’s files at the top level. How do I do this?”

By default, the files contained in an asdf module go in a subdirectory with the same name as the module. However, this can be overridden by adding a :pathname "" argument to the module description. For example, here is how it could be done in the spatial-trees ASDF system definition for ASDF 2:

(asdf:defsystem :spatial-trees
  :components
  ((:module base
            :pathname ""
            :components
            ((:file "package")
             (:file "basedefs" :depends-on ("package"))
             (:file "rectangles" :depends-on ("package"))))
   (:module tree-impls
            :depends-on (base)
            :pathname ""
            :components
            ((:file "r-trees")
             (:file "greene-trees" :depends-on ("r-trees"))
             (:file "rstar-trees" :depends-on ("r-trees"))
             (:file "rplus-trees" :depends-on ("r-trees"))
             (:file "x-trees" :depends-on ("r-trees" "rstar-trees"))))
   (:module viz
            :depends-on (base)
            :pathname ""
            :components
            ((:static-file "spatial-tree-viz.lisp")))
   (:module tests
            :depends-on (base)
            :pathname ""
            :components
            ((:static-file "spatial-tree-test.lisp")))
   (:static-file "LICENCE")
   (:static-file "TODO")))

All of the files in the tree-impls module are at the top level, instead of in a tree-impls/ subdirectory.

Note that the argument to :pathname can be either a pathname object or a string. A pathname object can be constructed with the #p"foo/bar/" syntax, but this is discouraged because the results of parsing a namestring are not portable. A pathname can only be portably constructed with such syntax as #.(make-pathname :directory '(:relative "foo" "bar")), and similarly the current directory can only be portably specified as #.(make-pathname :directory '(:relative)). However, as of ASDF 2, you can portably use a string to denote a pathname. The string will be parsed as a /-separated path from the current directory, such that the empty string "" denotes the current directory, and "foo/bar" (no trailing / required in the case of modules) portably denotes the same subdirectory as above. When files are specified, the last /-separated component is interpreted either as the name component of a pathname (if the component class specifies a pathname type), or as a name component plus optional dot-separated type component (if the component class doesn’t specifies a pathname type).

12.5.5 How do I create a system definition where all the source files have a .cl extension?

First, create a new cl-source-file subclass that provides an initform for the type slot:

(defclass my-cl-source-file (cl-source-file)
   ((type :initform "cl")))

To support both ASDF 1 and ASDF 2, you may omit the above type slot definition and instead define:

(defmethod source-file-type ((f my-cl-source-file) (m module))
  (declare (ignorable f m))
  "cl")

Then make your system use this subclass in preference to the standard one:

(defsystem my-cl-system
  :default-component-class my-cl-source-file
   ....
)

We assume that these definitions are loaded into a package that uses ASDF.


Next: TODO list, Previous: Getting the latest version, Up: asdf: another system definition facility   [Contents][Index]