Inference of attributes is a huge part of D programming. D has admittedly quite a lot of at­trib­utes, and four categories of attributes are related to functions:

• memory safety – Includes @safe, @system, and @trusted.
• pure – functional purity means that a function cannot access shared or global data.
• nothrow – Whether a function can throw an Exception (note this does not include Error or other Throwable derivatives, see my other post on this).
• @nogc – Functions marked with this cannot allocate memory from the GC. This includes hidden allocations the compiler might insert.

This post isn’t really about those attributes, and if you want to learn more about them, I recommend reading the D language specification, and searching for information about them on the D official blog (I wrote a post myself on writing @trusted code).

What I want to talk about here is attribute inference. Because of the proliferation of these different attributes, and because D is a very generative-heavy programming language (templates, CTFE, etc), it can be quite awkward to properly attribute some functions. D’s solution to this is to infer attributes based on the code being compiled. This is limited to functions that the compiler knows it must always have the source code available in order to use. These include:

• auto returning functions
• template functions
• functions inside a template
• functions inside another function
• lambda functions

Notably missing here are regular functions. Why? Because a function can be declared separately from the definition via a function prototype. Also of note: class member functions, even if inside a class template, will not be inferred. Since non-template class member functions are virtual, those functions must be explicitly attributed.

So what happens when attributes are wrong? The answer is that the compiler tells you something like the error from this code:

void foo() {
}
void main() @nogc {
    foo();
}

Error: `@nogc` function `D main` cannot call non-@nogc function `foo`

This is the error message from trying to call an incorrectly marked function foo. This is easy to figure out and correct — just put @nogc on foo and call it a day.

But what happens when the function that’s incorrectly marked is hidden behind an inferred function?

void foo() {
}
void bar(alias f)() {
    f();
}
void main() @nogc {
    bar!foo();
}

Error: `@nogc` function `D main` cannot call non-@nogc function `onlineapp.bar!(foo).bar`

No mention here of the real problem: foo is not marked @nogc. All we have is a reference to bar!foo. Now, this also isn’t too difficult to figure out, but this is also not the worst case. When inference failure happens, sometimes there are several layers to the problem. The function that needs attribution might be buried under 10 levels of templates, and maybe in those inside a static foreach, making it hard to figure out what, exactly, is causing the inference to do what it did.

So how do you find the problem? You do it by digging down through each layer until it becomes clear which part the compiler has seen that causes the inference to fail.

I’m going to pick one attribute — @nogc — and show how it works for that. But realistically, all of them can be done the same.

Technique 1: Explicitly mark the template

It’s not usually a good idea to mark a template with an attribute that can be inferred. Especially the attribute @trusted. But in this case, it is a temporary situation, where we want the compiler to dig a bit lower. You mark the template, and then when you solve the complete problem, you unmark it. To remind myself, I usually comment out the original line of code, and put a TODO: marker in there to remind me to remove it later.

If we mark our template above, we get a better error message:

void foo() {
}
void bar(alias f)() @nogc {
    f();
}
void main() @nogc {
    bar!foo();
}

Error: `@nogc` function `onlineapp.bar!(foo).bar` cannot call non-@nogc function `onlineapp.foo`

Nice! now we have the error that shows us the real problem — foo is not marked. Just mark foo, verify that it compiles, and remove the extra attribute from bar, done!

Technique 2: Copy and Rewrite

The problem with the first technique is that it sometimes adds failures for the rest of your code. What if we have something like this?

void foo() {
}
void bar(alias f)() @nogc {
    f();
}
void main() @nogc {
    bar!foo();
}

int x;
void allocateit() {
    x = new int(42); // actually uses GC
}
void otherFunc() { // not @nogc
    bar!allocateit();
}

Now we get two errors — If we are lucky! And in the case of allocateit, it really isn’t @nogc, so the marking of bar isn’t valid. In this case, we only want to mark bar as @nogc if the f parameter is @nogc. This is the main point of inference!

To fix this, we need to copy bar, add the expected attribute, and use the copy only when we are making the problematic call.

void bar(alias f)() { // leave this one alone
    f();
}
void bar2(alias f)() @nogc { // copy and add attribute 
    f();
}
void main() @nogc {
    bar2!foo(); // we get the correct error here
}
void otherFunc() { // not @nogc
    bar!allocateit(); // now this succeeds
}

In this way, we have isolated the path of the compiler for this one case, because this is the case we are interested in. We leave all other cases alone. In a large application where a template might be used in many places, this technique is essential.

Technique 3: Use static if

static if can help us make different decisions based compile-time data. Let’s say, for instance, the offending call is done inside a static loop. Maybe the template succeeds in being @nogc for some parameters, but not for others. Whether you use the normal path or the special attributed path has to depend on compile-time data detectible on the parameter.

This can be tricky, and there’s no “right” way to do this. It highly depends on what the “thing” is that triggers the error. I sometimes use type names, sometimes I use is expressions, sometimes I use __traits(compiles), etc. Whatever you use, single out the path you want to test, and make a specialized case for that one call.

void complicated(Args...)() {
    static foreach(T; Args) {
        static if(is(T == int)) bar2!T(); // specialized attributed path
        else bar!T(); // regular path
    }
}

Doing a full dig

Now that we’ve seen these techniques, how do we apply them to a real nasty 10-layer problem? In that case, we peel the rotten onion all the way to the core (likely caused by your missing attribute). Use whichever technique is appropriate at the next layer, and then repeat the sequence. Always look at the most inner inferred attribute function. Eventually, you will get to the answer.

This can be troublesome, since you may not control much of the code that is involved. Some of it may even be in D’s standard library! But don’t be afraid to (temporarily) modify your copy — none of the issues that might arise from doing this matter until the compilation succeeds. And at that point, you undo all the instrumentation.

Sometimes, I make a complete copy of the code, or just re-install the package once I’m done finding the problem. Don’t be afraid to take things apart, just remember which screws went to which parts!

Recursive instantiation inference failure

Sometimes, if a template is determined to depend on itself in certain way, the compiler gives up inference, and just assumes the worst case. An example:

auto forward(alias fn, Args...)(Args args) {
    return fn(args);
}
T factorial(T)(T val) {
    if(val == 1)
        return val;
    return forward!factorial(val - 1) * val;
}
void main() @nogc {
    auto x = factorial(5);
}

Error: `@nogc` function `D main` cannot call non-@nogc function `onlineapp.factorial!int.factorial`

Using technique 1, you can add @nogc to factorial, and it actually just compiles!

Unfortunately, there is no simple fix here. You can mark factorial explicitly @nogc, but this means that if some T value uses the GC, it can’t be used with factorial. These can sometimes be the hardest to diagnose, since normal techniques do not work.

I’ve seen different approaches to this, including using introspection to apply explicit attributes (which is not an easy thing to do). It may involve simply dictating to users the required attributes, and if you don’t use them, you don’t get to use the library.

I would like to see the compiler just become smarter about this. I believe that it could try compiling with the most restrictive attributes, and it should work most of the time. There might be some pathological cases that prevent inference, but just giving up is worse.

Great changes on the horizon!

In a recent version of the compiler (version 2.101.0), @safe inference has been instrumented so that when an inference results in failed compilation, the compiler does a lot of this work for you! Let’s take our original example, and replace @nogc with @safe (and compile with 2.101.0 or later)

void foo() {
}
void bar(alias f)() {
    f();
}
void main() @safe {
    bar!foo();
}

Error: `@safe` function `D main` cannot call `@system` function `testsafe.bar!(foo).bar`
       which calls `testsafe.foo`

That second error message is saying that the call to foo itself is actually what makes that instantiation of bar unsafe. We no longer have to instrument bar! Imagine that this is a call chain that is 7 layers deep. Having the compiler explain each layer without having to instrument it is going to save a lot of time.

Unfortunately, this is only for @safe code, and not for any of the other 3 attributes. Hopefully these improvements will be mimicked for all attributes, and instrumenting code will be a thing of the past!

But until then, hopefully this post helps you find some of these nasty inference bugs without too much hair-pulling!

D1 style operator overloads have been deprecated in D2 since version 2.088, released in 2019. Version 2.100, released last month, saw those operator overloads removed completely from the language. However, using D’s fabulous metaprogramming capability, it is possible to write a mixin template shim that will allow your D1 style operator overloads to keep working.

For sure, the best path forward is to switch to the new style of operator overloads. But there can be good reasons to keep using the old ones. Maybe you really love the simplicity of them. Maybe you use them already for virtual functions in classes, and don’t want to change. Maybe you just don’t want to do much code editing to an old project.

Whatever the reason, this post will show you how to do it easily and succinctly!

D1 Operator Overloads vs. D2 Operator Overloads

An operator overload is a way for a custom type to handle operators (e.g. + and -). In D1 these were handled using plain named functions, such as opAdd for addition or opMul for multiplication. For an example to work with, here is a struct type that uses an integer to represent its internal state:

struct S {
   int x;
   S opAdd(S other) {
      return S(x + other.x);
   }
   S opSub(S other) {
      return S(x - other.x);
   }
   S opMul(S other) {
      return S(x * other.x);
   }
   S opDiv(S other) {
      assert(other.x != 0, "divide by zero!");
      return S(x / other.x);
   }
}

void main() {
   S s1 = S(6);
   S s2 = S(3);
   assert(s1 + s2 == S( 9));
   assert(s1 - s2 == S( 3));
   assert(s1 * s2 == S(18));
   assert(s1 / s2 == S( 2));
}

Note how repetitive the operator code is! Plus, we only handled 4 operations. There are actually 11 math and bitwise binary (2-arg) operations that could be potentially overloaded for an integer. This doesn’t count unary operations (e.g. S s3 = -s1) or operations where S is on the right side of the op, with maybe an int on the left side (e.g. opAdd_r, opMul_r). If we needed to overload based on operand type, we could branch out into template functions, but that might not be that much less code.

D2 decided that a better way to handle bulk operations would be to use templates in order to handle operators. Instead of calling opAdd for + and opMul for *, it will call opBinary!"+" and opBinary!"*" respectively. This means we can handle all the operations in one function. To process them all, we can rewrite S like this:

struct S {
   int x;
   S opBinary(string op)(S other) {
      static if(op == "/" || op == "%")
         assert(other.x != 0, "divide by zero!");
      return mixin("S(x ", op, " other.x)");
   } 
}

void main() {
   S s1 = S(6);
   S s2 = S(3);
   assert( s1 + s2  == S( 9));
   assert( s1 - s2  == S( 3));
   assert( s1 * s2  == S(18));
   assert( s1 / s2  == S( 2));
   assert( s1 % s2  == S( 0));
   assert((s1 | s2) == S( 7));
   // and so on
}

Note how we not only have only one function (with a slight difference for the division operators), but we handle all math operations! The code is easier to write, less error prone, and less verbose.

Aliasing Operators

But what if you already have operators in D1 style, and you don’t want to change them, or merge them into one super-function?

D allows you to alias member functions to another symbol, and opBinary is no exception. Here is the original type, but with aliases for each of the operators:

struct S {
   int x;
   S opAdd(S other) {
      return S(x + other.x);
   }
   S opSub(S other) {
      return S(x - other.x);
   }
   S opMul(S other) {
      return S(x * other.x);
   }
   S opDiv(S other) {
      assert(other.x != 0, "divide by zero!");
      return S(x / other.x);
   }

   alias opBinary(op : "+") = opAdd;
   alias opBinary(op : "-") = opSub;
   alias opBinary(op : "*") = opMul;
   alias opBinary(op : "/") = opDiv;
}

Note that we are using a few cool features of D metaprogramming here. The aliases are eponymous templates which means I don’t have to write out the template long form, and we are using template parameter specialization to avoid having to use a single template and look for the covered operations inside the template, or having to use template constraints to filter out the operations we cover.

But we can do even better than this! Nobody wants to write this boilerplate code tailored to each type which may not all cover the same exact operators.

Mixin Templates

A mixin template is a template with a set of declarations in it. Wherever you mixin that template, it’s (almost) as if you typed all those declarations directly. Using the power of D’s compile-time introspection, it’s possible to handle every single possible operator overload that D1 could offer, by writing aliases to the D1 style operator overload, automatically.

In order to do this, we are going to have three rules. First is that we don’t care if the operators are properly written in D1 style. As long as the names match, we will forward to them. We also don’t need to worry about overloads based on the types or parameters accepted, as aliases are just name rewrites. Second, this mixin MUST be added at the end of the type, because otherwise, the entire type’s members may not have been analyzed by the compiler (this may change in a future version of D). Third, D does not allow overloads between the mixed-in functions and regular functions — the regular functions will take precedence. So you cannot define any D2 style operators of a specific name (e.g. opBinary). If you want D2 operators, convert the whole thing, don’t use some D1 and some D2.

Let’s write just the opAdd declaration in a mixin template, and see how it works.

mixin template D1Ops() {
   static if(__traits(hasMember, typeof(this), "opAdd"))
      alias opBinary(op : "+") = opAdd;
}

There’s a lot of meta code in here, I’ll explain it all.

The mixin template declaration is telling the compiler that this is a template specifically for mixins. Technically, you can use any template for mixins, but declaring it a mixin template requires that it’s only used in that way.

If you don’t know what static if is, I highly recommend reading a tutorial on D metaprogramming, as it’s essential for almost every metaprogramming task. Needless to say, the contained code is only included if the condition is true.

__traits(hasMember, T, "opAdd") is a specialized condition that is true only if the specified type T (in this case, the type of the struct the mixin is being added to) contains a member having the name opAdd.

And finally, the alias is as we wrote before.

Now, how would we use this inside our type?

struct S {
   int x;
   S opAdd(S other) {
      return S(x + other.x);
   }
   S opSub(S other) {
      return S(x - other.x);
   }
   S opMul(S other) {
      return S(x * other.x);
   }
   S opDiv(S other) {
      assert(other.x != 0, "divide by zero!");
      return S(x / other.x);
   }

   mixin D1Ops;
}

That’s it! Now opAdd is hooked via the aliased opBinary instead of via the D1 operator overload. Therefore, S + S will compile on 2.100 and later. However, the other operator overloads will not.

Why do it this way? As we will see, using the static if allows us to mixin the template regardless of whether opAdd is present or not. Using this feature, we can handle every possible situation with regards to existing operator overloads.

Using the Full Power of D

Adding each and every operator overload to the mixin is going to be very repetitive. But there is no need to do this, D is a superpower in metaprogramming! All we need to do is lay out the operation mappings, and we can use another specialized metaprogramming feature, static foreach, to avoid having to repeat the same boilerplate over and over.

With this, we can handle every binary operation that the struct might have written D1 style:

mixin template D1Ops() {
   static foreach(op, d1;
     ["+" : "opAdd", "-" : "opSub", "*" : "opMul", "/" : "opDiv",
      "%" : "opMod"]) {
      static if(__traits(hasMember, typeof(this), d1))
         alias opBinary(string s : op) = mixin(d1);
   }
}

Let’s look at the new things we have added to the mixin template. The first thing is an associative array of string to string, indicating which ops should map to which D1 function names. static foreach is a feature which will, at compile time, loop over all the elements in a thing that normally you would iterate at runtime (in this case, the associative array). It’s as if you wrote all those things out directly one at a time, with the symbols op and d1 mapped to the keys and values of the associative array containing the operation mappings.

See how our static if has changed a bit, instead of using a string literal, we use the d1 symbol, which in the first loop is "opAdd", in the second loop is "opSub" and so on.

In addition, there is one minor change in the alias. Because we must alias the opBinary call to a symbol, and not a string, we must fetch the symbol based on its string name. mixin(d1) does this. This is a relatively new feature, in older compilers we could still achieve this with a single mixin statement for the whole alias statement, but just calling mixin on d1 is a lot cleaner looking.

With that, our final code looks like this:

mixin template D1Ops() {
   static foreach(op, d1;
     ["+" : "opAdd", "-" : "opSub", "*" : "opMul", "/" : "opDiv",
      "%" : "opMod"]) {
      static if(__traits(hasMember, typeof(this), d1))
         alias opBinary(string s : op) = mixin(d1);
   }
}

struct S {
   int x;
   S opAdd(S other) {
      return S(x + other.x);
   }
   S opSub(S other) {
      return S(x - other.x);
   }
   S opMul(S other) {
      return S(x * other.x);
   }
   S opDiv(S other) {
      assert(other.x != 0, "divide by zero!");
      return S(x / other.x);
   }

   mixin D1Ops;
}

void main() {
   S s1 = S(6);
   S s2 = S(3);
   assert( s1 + s2  == S( 9));
   assert( s1 - s2  == S( 3));
   assert( s1 * s2  == S(18));
   assert( s1 / s2  == S( 2));
}

You’ll notice that I intentionally included opMod in the mixin, even though our type does not have it. This demonstrates the power of the static if to only provide aliases if the appropriate D1 operator overload exists.

Filling it out

All that is left for opBinary is to fill out the mappings to handle any possible existing D1 binary operations. As long as you have a D1-style operator, the mixin will generate an alias to cover it.

And finally, any other D1 style operations as listed in the changelog, such as opUnary or opBinaryRight can also be covered by adding another loop. You could even nest the mappings if you wanted to, or include the name of the template to alias as part of the mapping. Or you might notice that all the opBinaryRight operators are the same as the opBinary operators (except in), and just do both at the same time.

You also might not using static foreach for this, and actually write them all out by hand, simply because static foreach is slightly expensive, and so is constructing an associative array at compile-time. Remember, once this template is done, there will never need to be any updates to it. The advantage of using a loop is you have to write a lot less code, which makes it a lot less error prone.

And if you aren’t in the mood to do it yourself, here is a gist mapping the entire suite of D1 operator overloads.