- hySite

It is beginning of June and it means, that time for first post of the year has become. Enum flags are useful wrapper around bitmasks and can be used to store several flags into one field/property. Internally they are just integers where each bit represent a flag state and can be set on or off. For example, lets define what next weekend should look like by writing down possible plans:

Here each state have their own 1 bit in place and can be combined with every other one. By adding Flags attribute we tell .net to treat this enum as a bitset.

[ApiController]
[Route("[controller]")]
public class WeekendController : ControllerBase
{
	[HttpGet]
  public WeekendState Get() =>
	WeekendState.DRUGS | WeekendState.DONUTS;
}

If we call this controller we will get 18. Donuts is 10000 aka 16, Drugs is 10 aka 2. 2 + 16 = 18, easy. But if you don’t see backend code it can be not so easy. I’m asking second main question of the universe - what should I do on weekend and I’m getting 18? What doesn’t this mean? ¹

We need to serialise enum flags as strings. Easiest way to do that - add json serialised options in Startup.

services
    .AddControllers()
    .AddJsonOptions(options => 
        options
            .JsonSerializerOptions
            .Converters.Add(new JsonStringEnumConverter())
        );

Here JsonStringEnumConverter is default way of enum serialization. After these changes our little endpoint will return “DRUGS, DONUTS” which is an improvement. But frontend (or whoever will use api) may not like string with separated with comma values. The most logical way would be to return the array. To do that we need to write our own JsonConverter.

public class FlagJsonConverter<T> : JsonConverter<T> where T : struct, Enum
{
    public override T Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        //TODO
    }

    public override void Write(Utf8JsonWriter writer, T value, JsonSerializerOptions options)
    {
        //TODO
    }
}

services
    .AddControllers()
    .AddJsonOptions(options => 
    {
        options
            .JsonSerializerOptions
            .Converters.Add(new FlagJsonConverter<WeekendState>());
        options
            .JsonSerializerOptions
            .Converters.Add(new JsonStringEnumConverter());
    });

First method in our new converter is Read, which takes values from Utf8JsonReader and does everything possible to convert them into required enum (converter class is generic and can be used for all enums).

var derserialised = JsonSerializer.Deserialize<string[]>(ref reader);
var textValue = string.Join(',', derserialised ?? new string[0]);

return Enum.TryParse<T>(textValue, ignoreCase: true, out T result)
        ? result
        : (T)Enum.Parse(typeToConvert, "0");
}

Input string for that is parsed json token, which we convert to string array (if possible) and then join back as comma-separated strings. May be there are more elegant solution, but this was a fastest to code. And then this string is parsed using default parser. If failed - it tries to parse zero value, which lead to simplest result (WeekendState.None).

writer?.WriteStartArray();
var values = Enum.GetValues<T>().Where(e => value.HasFlag(e));
foreach (var val in values)
        writer.WriteStringValue(val.ToString());
writer?.WriteEndArray();
}

This is taking existing value, iterate it into all assigned flags and convert them into strings. And, on the way wrap it into array.

[Theory]
[InlineData("[\"DONUTS\"]", WeekendState.DONUTS)]
[InlineData("[\"ROCK\", \"ROLL\"]", WeekendState.ROCK|WeekendState.ROLL)]
[InlineData("[]", WeekendState.NONE)]
public void TestDeserialisation(string input, WeekendState result)
{
    var utf8JsonReader = new Utf8JsonReader(Encoding.UTF8.GetBytes(input), false, new JsonReaderState(new JsonReaderOptions()));
    var converter = new FlagJsonConverter<WeekendState>();
    var value = converter.Read(ref utf8JsonReader, typeof(WeekendState), new JsonSerializerOptions());
    Assert.Equal(value, result);
}

Bonus: spending each weekend eating donuts and drugs can be tedious. Let’s update controller code:

[HttpGet]
public WeekendState Get() 
{
    Random rnd = new Random();
    var shuffled = Enum.GetValues<WeekendState>().OrderBy(c => rnd.Next());
    var values = shuffled.Take(rnd.Next(shuffled.Count()));
    var result = WeekendState.NONE;
    foreach (var val in values)
        result |= val;
    return result;
}

It always contain random amount of random items. It also always include NONE value, which is not a proper solution.

Probably, the Answer to the Ultimate Question of Life, the Universe, and Everything is three parts: 32 + 8 + 2, we just don’t know the name of the proper Enum.↩

HexThrees

Today I'm releasing my first app for iOS, that I have developed: HexThrees!. This is a small puzzle game, that was inspired greatly by 2048 and Threes!. I have played 2048 for a long time in the past and one day a question came into my mind: what would happen, if a game field would be a hexagon? Overall, hexagons are Bestagones! This ideas were collided with fibonacci numbers, that we have used in kanban planning weekly at work.

I wanted to create simple, easy mind game, that can keep my hands busy, when I listen to podcast or talk by phone. Merging rules might look overcomplicated at first glance, but after a while I found them much more interesting and fun to play with comparing to 2048 and Threes!
screenshot

Although initial commit was made in april 2018, there were big gaps in development due to other projects, family, my real job and lazyness. Most of the game logic, graphic effects and base functionality was written in a train during commute between small town where I live and Hamburg where I work. But 2020 finally gave me opportunity to finish this game, polish it (as far as time allows me) and publish.

And this last mile was especially rough. I knew, that last percent takes more time than any other percent, but this feels especially frustrating because I kept promising to myself to get rid of this project as fast as I could and not waste more time on a hobby, that will not bring any amount of money. But still, the least stuff was left to do, the more it seems it took time. Iap integration, l10n (only english and russian), stupid bug in tutorial¹, screenshots for all sizes (hello, hardcoded save!), preview video (hello, ffmpeg!), first app store rejection and more small issues. Tasks, that I estimated for couple hours, required a day, and when I though, that I can deal with all in a couple days, it actually took a couple weeks.

But now it's over. All I left to do is to finish this post, push "publish" button, write some messages for reddit and relay.fm discord, where someone might be interested in that kind of game, publish instagram post and relax for a while. Hopefully, there would be no significant bugs².

This is a weird year. I (kinda) lost my main job, run 5k for the first time in summer, then 15k a month ago, which would be totally insane for me a year ago, and now this. I was writing small game prototypes since 2012³ and always wanted to release something for real. Today is the day.

Don't use SKAction:repeatForever, if you can help it. When user open control center, timer stops and on resume it multiplies several time. I.e. if timer should repeat every second, and game was suspended for 3 seconds, then on resume it will fire three times every second.↩
Famous last words.↩
My first 'games' were written in qBasic in 8th grade (around 2002), but that shit can't count.↩

Json to classes

Just a little thing, that I learned today. Here is the usual example: someone gave you example json for some API, that you need to implement and first step would be – created to classes for this json. There is a nice feature in Visual Studio: copy json in clipboard, then you can Edit -> Paste Special-> Paste JSON as classes.
vs paste json as class

One problem for me was that all fields were lowercase. Fixing them manually was too boring, so I wrote regex to replace first letter to uppercase: (\bint|\bbool|\bstring|\bfloat\bDateTime) ([a-z]) <- replace this
$1 \u$2 <- to this
vscode replace regex

No sure, that it will be useful for someone, I’m just proud of myself each time when I can use regex 🤓

If you are reading this, that mean that I have ..

..deployed this blog to azure web apps successfully. But it was a bumpy ride there, let me tell you.

how it was before

I have started this blog as a sandbox project and first things that I have learned in 2017 were: how to build docker image, push this image to repository and how to build dotnet core application. Therefore, every new version that I build, I started manually this scripts:

And after that I have to ssh to linode virtual linux and run third script there:

I think everything here is self is self-explanatory¹ and easy enough to be written by a person, who is started to looking ways to develop simple applications into the cloud. Deployment of new version was a cumbersome, but reliable path of running all three of these scripts together.

Goal

What I was looking is service, that can take docker file, access to repository, restore, build, publish and create docker image, that will contain latest version, then deploy it into some cloud – and all of this should happen automatically, on new commit in master and without any manual steps. Ideally, it also should beforehand check, that tests are running, builds are building and notify me, if something is wrong.

Dead ends

Because I use only one image, I ignored all solutions, that have “Kubernetes” in name. Dunno, it just scared me a little, I was looking for a simple ways to start. I knew, that there are two major cloud operators: Azure and AWS and for my purposes I decided to stick with azure. They have some new service, called container instances, that is used exactly for what I need – take docker file, build image, run it and don’t ask any more questions.
Besides azure portal, MS have product “devops pipelines”, which purpose is still little unclear for me. I have created pipeline, that took repository, build my project and run it in container instance - but I met a problem.
All posts in this site stored in private GitHub repository and first thing, that hyston.blog is doing on start – is attempt to sign in into GitHub and load all posts. To do this, obviously, I need to store my GitHub credentials. When I build site on local machine, these username/password were stored in web.Production.config, which was gitignored. It is possible to add azure pipeline secrets, that would contain GitHub username and password and pass them into docker environment variables, but then they will be stored as linux environment variables, which is considered as insecure. There is other option to do this, that I’ll describe late, but by then I decided, that is a stop sign for me to use container instance. The other reason, that pushed me away from container instances was this comment in Stackoverflow:

Azure Web app

At first, I have decided created webapp with deployment directly code – so I should note care about docker at all. There is “deployment configuration” option in web app, that (if I remember correctly, it is using devops pipelines) should work as “wizard”. For me it refused to login into my GitHub account at first². After a day or two this issue has gone and I was able to create deploy my code directly in couple clicks. But there appears another problem: there is no way to choose, which linux distribution will be used to deploy my code. For GitHub integration, my blog is using LibGit2Sharp, which in turn, using very specific linux git library, that is exist only on Ubuntu LTS. Some details about this issue can be found here. For docker image, I just add ‘-bionic’ to runtime dotnet image and then it runs in ubuntu. But for automatic deployment configuration base image was Debian and my site throwing an exception on trying to use libgit2sharp. At this point, I was frustrated about Azure portal functionality overall and confused with Azure devops pipelines (as I understood, these are two different services, that use different accounts). So I went further into GitHub Actions.

Github actions

There are many ways and possibilities, that are opening, when you are using them. Simplest way is to build dotnet core app directly and deploy binary somewhere. Because I already added build layer to my docker file, I left compilation part there, although I dont like this solution now – wherever I deploy it now, it doesn’t use cache from previous deployments. It is possible to set my own machine as action environment, but I currently don’t have any computer that will run all the time.

Azure KeyVault

I’ll show my deployment script later, but before that I still have one unresolved problem: I do need GitHub credentials in runtime and I still don’t want to store them in open source configuration. Because I stick with azure, I decided to use azure vault. It is overloaded and cumbersome service (as everything with Microsoft), but it ended up well documented (probably, even overdocumented, because I have found several very similar tutorials) and with this tutorial I have added support for it in my code. When I run locally, it take azure credentials from my machine (because I have installed and logged in azure cli) and in docker I have added environment variables, that imported from arguments. This is more secure, than just storing plain credentials in environment, because with azure client id this can be retained only from azure web app. And to pass them into dockerfile, I store them in GitHub secrets, which is inaccessible from view. So, to recap:

It may sound overcomplicated (that was stopped me from using it for a long time), but in hindsight this is most secure solution, as I can see.

Github actions (again)

To finish my GitHub action I needed only one step: deploy builded image. After all issues that I have seen trying to create devops pipelines, this was easy.

Happy end

And that’s basically it. It took me more than one hour to write this post about a thing, that took me almost a month to figure out ⁴. In the end it is just two scripts: Dockerfile, deploy.yaml created azure account and several settings in azure portal and GitHub. I hope, that in a year, when I will forget about all of that and would need to implement similar stuff, this blog post will help me to refresh my memory.

P.s. Have I mentioned, how I hate yaml indent-based structure? 🤯 This is good in theory and keep structure organised, but I have made too many errors during my development adventures.

I know, that using separate Service variable and removing all images is redundant, as well as other missing improvements. I just copied my own 3yo script↩
Even that my microsoft account is directly bonded to my GitHub account.↩
I made a mistake here which left unnoticed for too long time and I have spent a day, trying to figure out, why I don’t have permission to pull image, before I realised that image name was wrong.↩
Keep in mind, this is a hobby project during summer, I didn’t spent actual month of working time to do this 😛↩

Using C# Reflection and Expressions for refactoring

disclaimer: here is example of refectoring, that is not optimised and well polished. With more experience I'll find better way to achieve what is written here. I have heavy feeling, this is the example of code, that I'll look back N years later with only one thought: "who, the ..., wrote this??"
However, I want to share this experience anyway to have a reference later.

Just imagine situation: you have 40+ classes with same interface and all of them have function with same signature and almost same body except one generic parameter, that is class type itself.

interface IBaseJob
{
    void Run(IJobCancellationToken token);
}

public class ConcreteJob : IBaseJob
{
    public void Run(IJobCancellationToken token)
    {
        /* implementation */
    }

    public Task Execute() =>
        BackgroundJob.Enqueue<ConcreteJob>(job => job.Run(JobCancellationToken.Null));
}

I usually remove unnecesarry classes and types from examples, but these classes from Hangfire library would be important later.

Here I dont like Execute function, that is repeated in several dosen other classes with only small difference:

That sounds like a opportunity to move some functionality into base class. In C#8 it is possible to add interface default implementation, but we still use C# 7.3, so all I could do is to replace interface with abstract class.

public abstract class IBaseJob
{
    public abstract void Run(IJobCancellationToken token);

    public Task Execute(IJobExecutionContext context)
    {
        //@todo: call here BackgroundJob.Enqueue<ThisJobType>(job => job.Run(JobCancellationToken.Null));
    }
}

...and all duplicated Execute functions from all jobs can be thrown away! 🥳 However, we still have a small problem here: How to call Enqueue method properly? We cannot call it directly, because we are not aware of class type in compile time, so we need to construct this function.

Well-experienced C# developer have access to types, methods and all stuff, that was used to write code, that he is currently writing 🤪 Start with type Type, that describes type. Enqueue is a static function from BackgroundJob, therefore we need this type and get list of methods, that it supports:

At first I thought just to get method by name (.GetMethod("Enqueue")), but there are 4 methods with same name and friendly exception kindly asked me to be less ambiguously. This is a common problem with reflection code: you usually doesnt know if something is wrong in compile time, all errors show itself only when you actually run the code.
So, method, that I needed, is 1) generic and 2) have Expression<Action<T>> as argument. What is an Expression and why they need Action I will explain later.
GetMethod return MethodInfo, class, that describes all information about concrete method. Then we need to call GetParameters to, well, get method parameters. I'll probably better explain it by adding comments into code:

var enqueueMethod = typeof(BackgroundJob)
    .GetMethods() // We have all BackgroundJob methods here
    .Where(m =>
        m.Name == "Enqueue" &&
        m.IsGenericMethod &&
        m.GetParameters().Length == 1) // Filter out only generic, with proper name and one parameter
    .Select(m => new
    {
        method = m,
        paramType = m.GetParameters().Single().ParameterType
    })  // We need to have a paramter type for further investigation and same MethodInfo in tuple to return it later
    .Where(p =>
        p.paramType.GetGenericTypeDefinition() == typeof(Expression<>) &&   // Generic type of argument should be 'Expression<>'
        p.paramType.GetGenericArguments().Length == 1 &&
        p.paramType.GetGenericArguments().First().GetGenericTypeDefinition() == typeof(Action<>)) // We need to go deeper and find generic arguments from generic arguments 🧐
    .Select(p => p.method) // If single parameter has type Expression<Action<>>, we are good
    .Single()   // An I'm sure, that there is only one of them. However, that's a bad practice, if someone update Hangfire version and there would be another method, that will pass by this criteria

And now we have MethodInfo about what we need to call!
Can we can call it? No, not yet.
Remember that 'Action<T>' ? We need to place current job type there. Type can be found easily with GetType(). Difference with typeof is that later is used in runtime, unlike that GetType() user can place current type in compile type, that is what we needed. And then we construct proper method using MakeGenericMethod

Now we are talking!
But we still have one small detail - Enqueue method is not just genetic, it take generic argument. And this argument is itself a Expression that take Action as a generic argument, which broke my brain for a second, when I was trying to figure it out.🤯

This haven't done anything clearer for me. Expression is a smallest part of code, that can be described: variable, constant, function or methods argument. Let's say we have code line a + 3. We can represent it as expression, where a would be Expression.Variable, then 3 would be Expression.Constant and whole line would be Expression.Add with previous two as a parameters. Lambda itself can be presented as a Expression too¹. And now we need to build expression, that represent code job => job.Run(JobCancellationToken.Null) where job would be current type.

var cancellationTokenArgument = Expression.Constant(JobCancellationToken.Null, typeof(IJobCancellationToken));
var jobParameter = Expression.Parameter(concreteJobType, "job");
var runMethodInfo = concreteJobType.GetMethod("Run");
var callExpression = Expression.Call(jobParameter, runMethodInfo, cancellationTokenArgument);
var lambda = Expression.Lambda(callExpression, jobParameter);

So, finaly, we got everything: method, that we need to call and parameters to call it. We can do magic with MethodInfo.Invoke:

Whas it all worth it?

So, now we have one function in base class instead of many (many many) small functions in dosens of files. Does code became clearer? I doubt so. Does it faster? Definitly not ². We just followed DRY principle and removed duplicated code. I guess, most valulable benefit, that we gain is the experience in writing code, that works with Reflections and Expressions. In writing day-to-day buiseness logic I not often have a chance to use them 😋
We have another example for using reflections in our project: every time someone add new field to class, that represents possible user rights, startup tool make changes to database to add new fields in users table. Otherwise, Reflections and Expressions are used in more common system code like Linq, Automapper etc.

I'm also new to all this terminology. As I understood, 'lambda' mean usually 'lambda expression', but Action and Func types are called delegate types, which is already compiled code.↩
Expressions should be compiled to IL to run and its a costly process. Not sure, where exactly it is happening here, I guess, already on Hangfire side. Need to run time test for this particular case, but in general, using expression trees can cause performance loss.↩

JsonConverter and enum flags