Implement new Concept Exercise: macros

[angelikatyborska]

Learning objectives

• defmacro
• the Macro module
• what's an AST
• when not to use macros

Out of scope

I am not sure if explaining require/use in the same exercise would be a good idea. It's a lot of difficult concepts at once. But maybe it makes sense?

Concepts

• macros

Prerequisites

Come up with something that will put this concept far down the concept tree. Maybe this should depend on a non-existent yet concept of dynamically defining functions without macros? (see #583 (comment))

Practice exercises

Those practice exercises should have macros in their prerequisites and as practices:

• dot-dsl

[Scaletsang]

I am still learning Elixir, but I would like to help as not only I'm in love with functional programming (I started with Haskell and Lisp) but also impressed by how macros in elixir are really expressive. I want to share a few of my thoughts:

Macros are only used when you need to create a new interface for writing codes. In effect, the programmer who uses the macros does not need to know the AST underneath it but only the logo decided by the programmer who writes the macros. If I were to teach a lesson about macros, this is what I will say in order:

1. Revealing that we are always using macros. e.g. defmodule, and test macro in all the test files in every exercise.
2. AST in elixir. The easiest example would be a tuple representation of a function. Here very smoothly can we introduce quote and unquote. One can type in their iex quote do: 1 + 2 and see the AST of 1 + 2.
3. With this understanding, then there will be less ambiguity in learning defmacro, and also less misused because now it should be clear that they are messing with manipulating/ injecting values in an AST and thus only used when you are creating new syntax or creating complex applications.
4. Macros are executed at compilation time. Then introduce some effect of it:

• therefore you cannot have recursion at macros.
• If you want to use macros, you have to compile it first, and hence the require macro which compiles macros from other modules.
• use macro. It does 2 things: Require the macros from another module and run using/1 macro in the specified module to the current module. As a result, it injects code directly from using/1 macro to the current module. ( need to completed import concept first, so that they understand that import is just importing functions and macros, but use and require is about compiling macros from other modules)

So

1. quote&unquote: a little guide about AST, and how quote and unquote can inject values in the AST
2. macros: defmacro, Macro modules
3. require & use

Tell me what do you think!!

[angelikatyborska]

1. quote&unquote: a little guide about AST, and how quote and unquote can inject values in the AST
2. macros: defmacro, Macro modules
3. require & use

I'm thinking that those concepts are all so complex for a beginner that it might be best to have 3 separate exercises, one for each of those concepts. The goal of a concept exercise is to be solvable in 5 minutes by a developer fluent in Elixir.

I wonder if the first concept should be called "AST" and cover quote and unquote, or if "quote" and "unquote" should be two separate concepts? Does "unquote" even make any sense without macros?

Maybe the first concept should be about working with ASTs by creating them with quote, changing somehow (Macro.traverse), and then printing with Macro.to_string?

I assume you already worked through some of the concept exercises on exercism.org so you know that they're all split into tasks that usually deal with small pieces of functionality.

Usually we start designing exercises by coming up with a story and an example solution that fits the story. For example, an exercise for working with the AST (no defining macros yet) might first have the student just hardcode something that uses quote, and then define some small functions that modify the AST in some way, and then as the last task convert the AST to a string?

A story idea could be... "Top secret!" you have this code that uses the name "microblogging_platform" in function names and variable names. Now you need to share it with an external agency but you don't want to reveal your secret plans to write a better microblogging platform, so you need to find all function names, variable names, and module names to change "microblogging platform" to some code name ("flying octopus", "red bear", or something :)).

This idea is just to get you started in case you don't have inspiration. Feel free to come up with something different.

For the other two concepts (macros / require&use), I don't have any thoughts yet.

If you start experimenting around and writing something, make sure to open a draft PR early. Writing a full exercise 100% correctly with all the necessary documents takes a lot of time and I don't want you to have redo a lot of work in case something's wrong early on.

[angelikatyborska]

Just FYI, I tried out this idea and it doesn't work very well IMO:

A story idea could be... "Top secret!" you have this code that uses the name "microblogging_platform" in function names and variable names. Now you need to share it with an external agency but you don't want to reveal your secret plans to write a better microblogging platform, so you need to find all function names, variable names, and module names to change "microblogging platform" to some code name ("flying octopus", "red bear", or something :)).

It relies on pattern matching on module names, which look a bit weird in the AST because of the aliases:

quote do
  Foo.Bar.Baz
end
# => {:__aliases__, [alias: false], [:Foo, :Bar, :Baz]}

It also doesn't use an accumulator in Macro.prewalk and it doesn't care about prewalk vs postwalk. And then, at the end, the usage of Macro.to_string produces ugly output in some cases (empty block) and it will change its output in the next major Elixir release (see elixir-lang/elixir#11172).

Maybe a nicer exercise idea for working with the AST would be collecting some statistic/metadata about code instead of changing code, which would force the usage of an accumulator and wouldn't require Macro.to_string.

[jiegillet]

To stick with the spy theme, how about this?

Top secret! Your secret informer is an Elixir developer and is encoding secret messages in his code. If you take the first 2 letters of every function (public or private) in a module in order and concatenate them, you will get the secret message.

defmodule TotallyNotTopSecret do
  def force(mass, acceleration), do: mass * acceleration
  def uniform(from, to), do: rand.uniform(to - from) + from
  def data(%{metadata: metadata}), do: model(metadata)
  defp model(metadata), do: metadata |> less_data |> Enum.reverse() |> Enum.take(3)
  defp less_data(data), do: Enum.reject(data, &is_nil/1)
end

[angelikatyborska]

Your secret informer is an Elixir developer and is encoding secret messages in his code.

I love it 😂 it's perfect.

"found a mole" 😮 😱

[simpers]

Hello! Just joining the discussion with a little tip here as I had an idea that maybe could help!

I started learning how to use macros earlier this summer and one of my experiments was to actually solve RotationalCipher using a macro. The idea to me was that I wanted to have an API that allowed arbitrary alphabets as I like writing in Swedish, but I didn't want to rebuild the map I created each time you ran a function. So I created a CipherMap module that you could use – for example use CipherMap, "AaBbCcDdEeFfGgHhIiJjKk..." – and then it would validate this and it would build the map during compile time and add it as an attribute for you to use in your module later, and the original alphabet would be exposed through a generated function and so on.

I think it could be useful to revisit old problems and attack them from a macro angle instead, if they're suitable (with great power comes great responsibility and all that haah), for the introduction part of using metaprogramming in Elixir to solve something, so that once some of the foundations are there we could add content which would be best solved using some sort of DSL or something and go from there?

I am still however very new to metaprogramming and I myself have a lot to learn. Just thought about this when I stumbled upon this issue :)

EDIT: I realised I can link to my solution so you can see what it looks like if you want see it RotationalCipher solution

[angelikatyborska]

@simpers Hi 👋 thanks for joining the discussion!

I think it could be useful to revisit old problems and attack them from a macro angle instead, if they're suitable

One of the basic assumptions about learning exercises is that they are not supposed to get more difficult to solve as the concepts themselves get more advanced. The goal is still to have an exercise that a fluent Elixir developer will solve in 5-10 minutes. That disqualifies most of the existing practice exercises from getting turned into learning exercises 😞

[simpers]

Hello! 👋

I see, that makes sense. The core idea of my solution to that exercise was to have a compiled value based on the coder's input, so if we can take that concept and turn it into something then that would do the same thing. But for a new exercise then?

I did notice after commenting here that you had published an AST exercise a few days earlier so I will check that out too as soon as possible. 😄

[pul-paulsen]

Hello!

Is this concept exercise task still open after 2 years?

I haven't contributed anything to exercises yet, but would like to. I have done a little on meta-programming and just use it for optimizing my Roman Numerals solutions, see also my explanations in the comment section there.

Here is my idea for the exercise for dynamic definitions / metaprogramming without macros / use:

1. You are given an implementation, that uses some kind of algorithmic approach, which runs to slow since it is to naive (I had that problem with my first approach to Palindrome Products: I used a similar naive approach for the plalindrome test like most do, which show to be far too slow (I generally think it would be a good idea to add timed-out performance tests "f(x) does not take more than 500 ms", because many seem not to care about afficiency at all)
2. You are asked to figure out an approach that runs more efficiently by exploiting pattern matching and guards (I did that in my Palindrome Product solution
3. Since handcrafting those methods by hand you are asked to come up with a meta-programming solution, which creates the clauses algorithmically (like in the Roman Numerals example above) (we would need to analyze the source file in comparison to the compiled module in order to test, that the definitions were really done dynamically)

A second exercise (one would be too complex) would take that idea to the world of macros and use:

1. Given a source file, which has some code for defining a table like for the roman numerals (another idea would be a decision table for business rules), you are asked to make that table code compilable and working
2. You have to define a module, which implements the macros and can be added by used
3. You have to add the code generating based on that macro (like encode/decode functions with patterns from the defined table entries)

What do you think?

[angelikatyborska]

Hi! Yes, this is still open because the maintainers that created all the other learning exercises (jiegillet, neenjaw, and myself) ran out of steam and out of ideas 😉

Help would be much appreciated, but keep in mind those goals of learning (concept) exercises:

• an experienced Elixir developer can solve it easily in no more than 15 minutes
• it's split in minimum 2, ideally 3-5 small steps, each step is either independent from the previous step or builds on top of the previous step.

I have to admit that I don't fully understand your ideas (I might be too tired at the moment) so I'm not sure if they fulfill those requirements. When I designed the other exercises, I would always start by writing the expected solution. What's a piece of code that uses the concept and all of its most significant subconcepts, while also avoiding any unrelated complex topics? Once I have that, I try to force a theme onto it. How could a piece of code that shows off dynamic function definitions without macros look like? It should probably start with generating clauses of one function with a known name, but then also include generating functions with dynamic names. Could you write a snippet like that? When you write it, does it make you think of a theme?

[pul-paulsen]

Hi Angelika,

thanks for your response (and all your great work on the Elixir track!).

In general, I understand and agree with the scheme of the solutions. I just have to make myself familiar how the track is maintained here on GitHub. Is there something like a guideline or similar for beginning contributors?

I will ponder the ideas and come back with a proposal or a PR.
I just wanted to check, if the demand still exists.

Regards,
Pul

[angelikatyborska]

The contributing rules are described in the CONTRIBUTING.md file in this repository.

[pul-paulsen]

Yes, thanks. My question was a little dumb and unnecessary; shortly after posing it, I detected the answer myself and also studied the structure of the repo a little bit, already. I guess, I can easily work with that and come back with a draft-PR