Jekyll2022-01-30T21:17:54+00:00https://www.nathanvangheem.com/feed.xmlNathan Van Gheem’s BlogMusings on Coding, Python, Open Source and DesignProposal: Federated OpenAPI Gateway2021-04-19T10:30:00+00:002021-04-19T10:30:00+00:00https://www.nathanvangheem.com/posts/2021/04/19/federated-rest-api-gw<h1 id="proposal-federated-openapi-gateway">Proposal: Federated OpenAPI Gateway</h1>
<p>Apollo has this great project to build <a href="https://www.apollographql.com/docs/federation/">federated GraphQL services</a>.
It allows you to build many GraphQL services that get consolidated together into one service.
It also allows you to do something really neat–you can build you services
together so that some act as a sort of “foreign key” join on data.
It is a great tool to build de-coupled micro-services without
needing to build composition services.</p>
<p>Now, why isn’t there anything like this for OpenAPI/REST?</p>
<p>Imagine having many services defining different endpoints all behind a gateway
to combine and consolidate them into one cohesive service.</p>
<p><img src="/assets/posts/openapi-gw/openapi-gw-high-level.png" alt="High Level" /></p>
<p>Where, as long as each service provides their own OpenAPI definitions, the gateway will
present one cohesive API.</p>
<p>User Service:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"paths"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"/users/{user_id}"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"get"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"responses"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"200"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"content"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"application/json"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"schema"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"$ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"#/components/schemas/User"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"components"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"schemas"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"User"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"object"</span><span class="p">,</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Review Service:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"paths"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"/users/{user_id}"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"get"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"responses"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"200"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"content"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"application/json"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"schema"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"$ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"#/components/schemas/User"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"components"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"schemas"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"User"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"object"</span><span class="p">,</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"reviews"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"array"</span><span class="p">,</span><span class="w">
</span><span class="nl">"items"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"$ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"#/components/schemas/Review"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"Review"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"object"</span><span class="p">,</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"body"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="nl">"product"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"$ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"#/components/schemas/Product"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Product Service:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"paths"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"/products/{product_id}"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"get"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"responses"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"200"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"content"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"application/json"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"schema"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"$ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"#/components/schemas/Product"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"components"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"schemas"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"Product"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"object"</span><span class="p">,</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Which would output one OpenAPI and service that provides:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"paths"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"/users/{user_id}"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"get"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"responses"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"200"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"content"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"application/json"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"schema"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"$ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"#/components/schemas/User"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"/products/{product_id}"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"get"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"responses"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"200"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"content"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"application/json"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"schema"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"$ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"#/components/schemas/Product"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"components"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"schemas"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"User"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"object"</span><span class="p">,</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="nl">"reviews"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"array"</span><span class="p">,</span><span class="w">
</span><span class="nl">"items"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"$ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"#/components/schemas/Review"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"Review"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"object"</span><span class="p">,</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"body"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="nl">"product"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"$ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"#/components/schemas/Product"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"Product"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"object"</span><span class="p">,</span><span class="w">
</span><span class="nl">"properties"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>And a <code class="language-plaintext highlighter-rouge">GET</code> request to <code class="language-plaintext highlighter-rouge">/users/1</code> would then respond with:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"name": "Foo bar",
"reviews": [{
"body": "Mario rocks",
"product": {
"id": "1",
"name": "Nintendo"
}
}, {
"body": "Sonic is better",
"product": {
"id": "2",
"name": "Sego"
}
}]
}
</code></pre></div></div>
<p>Behind the scenes the following requests would then be made:</p>
<ul>
<li>User service: <code class="language-plaintext highlighter-rouge">GET /users/1</code></li>
<li>Reviews service: <code class="language-plaintext highlighter-rouge">GET /users/1</code></li>
<li>Products service: <code class="language-plaintext highlighter-rouge">GET /products/1</code> and <code class="language-plaintext highlighter-rouge">GET /products/2</code></li>
</ul>
<p>There are many different ways to build it; however, the above
examples are a simple way it could work.</p>
<h1 id="harder-problems">Harder problems</h1>
<ol>
<li>One of the advantages of GraphQL is you only get back what you ask for.
An approach for OpenAPI would probably need to implement some sort of
<code class="language-plaintext highlighter-rouge">?include=reviews,reviews.product</code> implementation in order to return the data you want. Otherwise, the fan-out could be quite bad.</li>
<li>GraphQL is built for running multiple operations at a time. This allows
developers to leverage data loader patterns to alleviate fan-out problems.
It’s possible similar multi-operation endpoints would need to be
available to downstream services in order to solve this.</li>
<li>Error handling. Federated GraphQL already has issues
with this(no http status codes); however, I can see issues where one
service responds correctly but another does not. What happens if
a “foreign” key does not return a value?</li>
</ol>
<h1 id="update">Update</h1>
<p>I’ve written a POC implementation of the above in Python and <a href="https://github.com/vangheem/federated-openapi-poc">put it on GitHub</a>.</p>
<p><img src="/assets/posts/openapi-gw/openapi-gw-postman.png" alt="Postman example" /></p>Proposal: Federated OpenAPI GatewayThe Exploitation of Open Source2021-01-31T22:30:00+00:002021-01-31T22:30:00+00:00https://www.nathanvangheem.com/posts/2021/01/31/exploitation-of-open-source<h1 id="the-exploitation-of-open-source">The Exploitation of Open Source</h1>
<p>At its best, open source enables people to be more productive by using code that is
free and open to all to use and contribute to.</p>
<p>On the other hand, Open Source can also simply be used as a mechanism to entice consumers–its just a
mechanism to get people to try the product. Once hooked on the product, maybe that’s where you find
out some features aren’t available for production. Or maybe after you’re a hooked customer for years,
the licensing will change on you. There was a recent example of this when ElasticSearch
<a href="https://anonymoushash.vmbrasseur.com/2021/01/14/elasticsearch-and-kibana-are-now-business-risks">changed its license to make it less open</a>.</p>
<h2 id="open-source-business-models">Open Source Business Models</h2>
<p>There are many different types of business models around open source software.</p>
<p><em>Corporate Open Source</em>: This model is a similar to trial software. The open source version is some hindered,
less useful version of the full, better, enterprise option. Companies that do this are confluent
with Kafka oriented products and Elastic with ElasticSearch. There is a range in how “Open Source” these
corporate business models are. Over time, the companies tend to change their licensing and way they
contribute to the core open source product in order to entice users onto paid options.</p>
<p><em>Enterprise support</em>: This model is one where you pay for support of using the open
source products; however, the whole package and product is completely open source. These types
of companies are more difficult to identify because they seem to have more competition and
are not always centralized into one company.</p>
<p><em>Open Source Services</em>: This model offers limited free services to open source projects but asks you to
pay if you’d like to use it on private projects. There are smaller examples of this however with
CI, coverage reporting, and testing services–they will be free for open source but need to pay for
on-prem or private repositories. The goal here is to entice open source engineers who will hopefully
bring their open source tooling experience to the private sector and encourage corporate to pay
for services.</p>
<h2 id="the-model-matters">The Model Matters</h2>
<p>Not all companies can be built on selling open source software; however, the way open source is
used matters in those that do. Who is the customer? Is the licensing and source just a way to trick people
into using the software only to be trapped in all the up-sells with a single company?</p>
<p>The <em>enterprise support</em> business model is the only one that doesn’t look at its user-base
in a manipulative way. It is also a model that can encourage more collaboration on the
software with other companies or independent developers. It is the model used by <a href="https://plone.org">Plone</a>–a
open source community I’ve been a part of for years.</p>
<p>The model that has been bothering me more lately is the <em>Open Source Services</em> model. This model
aims to entice open source developers into using them so they will hopefully eventually be
adopted in corporate.</p>
<p>GitHub is now a Microsoft product. We shouldn’t be naive about why Microsoft saw GitHub as
a valuable company to acquire.</p>
<h2 id="why-open-source-transaction-costs">Why Open Source: Transaction Costs</h2>
<p>There are many reasons you might want to write and use Open Source software. You might
be <a href="https://www.gnu.org/philosophy/philosophy.en.html">philosophically aligned with it like Richard Stallman</a>
or you might just enjoy collaborating with other people while building things.</p>
<p>I agree somewhat on philosophical idea that software should be open; however, I think I’m more inclined
to enjoy open source because of how much more productive we all are with it. I think we’re more productive
with it not only because we can consume software for free in our own projects but also because
we can read and learn from it.</p>
<p>Additionally, I like to look at open source the way Michael Munger looks at the sharing economy in his book, <a href="https://smile.amazon.com/Tomorrow-3-0-Transaction-Cambridge-Economics/dp/1108447341?sa-no-redirect=1">Tomorrow 3.0</a>. In it, Munger argues that it’s all about lowering transaction costs. It seems to me Open Source software
is the same way–we are lowering the transaction costs of building software by sharing code
for problems that have already been solved and building infrastructure to make it easier to share
and collaborate on it.</p>
<h2 id="building-open-source-services">Building Open Source Services</h2>
<p>I would love to see open source developers work on building better open services and tooling
so we aren’t forced into mixed corporate open source offerings that lock us in or are over
priced to be able to use privately.</p>
<p>One example of this recently is <a href="https://about.codecov.io/">codecov</a>. It is a really nice
tool that is free to use for open source but has a pretty hefty cost for private projects(especially on-prem).</p>
<p>It bothered me that something like that shouldn’t already be an open source package. It’s a simple
set of features that I’m interested in: diff coverage and PR integration.
Now, for a company that has any code private, the minimum charge to use it is $50,000.
That is not a small amount and probably crowds out all small companies and some medium sized companies.
Even if you are a large company, it takes an approval process to justify a $50,000 product. The price
and process involved in getting a priced product in corporate land increases the transaction cost a significant amount.</p>
<p>This is why I started the <a href="https://open-coverage.org/">Open Coverage</a> project. It is a MIT licensed code coverage
service. You can use it in any way you want–even has docker images. Very low transaction cost!</p>
<p><a href="https://open-coverage.org"><img src="https://open-coverage.org/logo/logo2-crp.png" width="300px" /></a></p>The Exploitation of Open SourceScaling Python Web Applications: AsyncIO vs Threads2019-06-11T02:30:00+00:002019-06-11T02:30:00+00:00https://www.nathanvangheem.com/posts/2019/06/11/scaling-python-web-applications<h1 id="it-started-with-a-tweet">It started with a tweet</h1>
<p>Recently, Mike Bayer tweeted:</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">I can't stress enough what a bad idea it is to build out a database application architecture on top of a non-blocking IO approach, that is, asyncio, eventlet, gevent, etc. 1/</p>— mike bayer (@zzzeek) <a href="https://twitter.com/zzzeek/status/1134461181530427394?ref_src=twsrc%5Etfw">May 31, 2019</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Needless to say, I strongly disagree.</p>
<p>While, “database application” is a pretty broad category to designate not using AsyncIO for,
I wrote a framework that lives in this category: <a href="https://github.com/plone/guillotina">Guillotina</a>.</p>
<p>Also, there are many great projects that are designed specifically for AsyncIO and seem to perform
extremely well–just look at all the things the <a href="https://github.com/magicstack">magicstack</a> people
are doing with <a href="https://edgedb.com/">EdgeDb</a>.</p>
<p>I had quite a few responses to the tweet and I’ll try to expand on those things in this post.</p>
<h1 id="cpu-bound">CPU Bound</h1>
<p>Mike stresses that AsyncIO breaks down when you have CPU bound code. I do not disagree. I have
always advised that you should never have CPU bound code in AsyncIO and if you do, make sure
to use <code class="language-plaintext highlighter-rouge">run_in_executor</code>. Even then, it’s not ideal and you should think about solving your
problem another way(queue, another services, etc).</p>
<h2 id="threads-can-be-cpu-bound-too">Threads can be CPU bound too</h2>
<p>The point I want to touch on here is that being CPU-bound is not an AsyncIO only problem. You
can be CPU-bound with threaded web applications as well.</p>
<p>Let’s see how a thread-based application performs vs an AsyncIO applications for CPU bound
request handling.</p>
<p>For the context of this post, we will be using this <a href="https://gist.github.com/vangheem/00666c9cf264f883db119cca6c59016e">loader script</a>. It defaults to running 100 requests concurrently.</p>
<p>Our simple CPU bound threaded app will be a flask app that does a CPU heavy task for
half a second on each request:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="o">@</span><span class="n">app</span><span class="p">.</span><span class="n">route</span><span class="p">(</span><span class="s">'/'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">cpu_bound</span><span class="p">():</span>
<span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="p">.</span><span class="n">time</span><span class="p">()</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="mi">9</span> <span class="o">*</span> <span class="mi">9</span>
<span class="k">if</span> <span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start</span><span class="p">)</span> <span class="o">></span> <span class="mf">0.5</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">return</span> <span class="s">'done'</span>
</code></pre></div></div>
<p>Run using <code class="language-plaintext highlighter-rouge">./bin/gunicorn -b localhost:8080 cpu_bound_flask:app</code></p>
<p>And the same app, done with aiohttp would look like this:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">aiohttp</span> <span class="kn">import</span> <span class="n">web</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">hello</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="p">.</span><span class="n">time</span><span class="p">()</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="mi">9</span> <span class="o">*</span> <span class="mi">9</span>
<span class="k">if</span> <span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start</span><span class="p">)</span> <span class="o">></span> <span class="mf">0.5</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">return</span> <span class="n">web</span><span class="p">.</span><span class="n">Response</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="s">"done"</span><span class="p">)</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">web</span><span class="p">.</span><span class="n">Application</span><span class="p">()</span>
<span class="n">app</span><span class="p">.</span><span class="n">add_routes</span><span class="p">([</span><span class="n">web</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'/'</span><span class="p">,</span> <span class="n">hello</span><span class="p">)])</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">web</span><span class="p">.</span><span class="n">run_app</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
</code></pre></div></div>
<p>And we’ll also provide an aiohttp version that uses threads
for the CPU bound code using <code class="language-plaintext highlighter-rouge">run_in_executor</code>:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">asyncio</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">from</span> <span class="nn">aiohttp</span> <span class="kn">import</span> <span class="n">web</span>
<span class="k">def</span> <span class="nf">cpu_bound</span><span class="p">():</span>
<span class="n">start</span> <span class="o">=</span> <span class="n">time</span><span class="p">.</span><span class="n">time</span><span class="p">()</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="mi">9</span> <span class="o">*</span> <span class="mi">9</span>
<span class="k">if</span> <span class="p">(</span><span class="n">time</span><span class="p">.</span><span class="n">time</span><span class="p">()</span> <span class="o">-</span> <span class="n">start</span><span class="p">)</span> <span class="o">></span> <span class="mf">0.5</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">hello</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="n">loop</span> <span class="o">=</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">get_event_loop</span><span class="p">()</span>
<span class="k">await</span> <span class="n">loop</span><span class="p">.</span><span class="n">run_in_executor</span><span class="p">(</span><span class="bp">None</span><span class="p">,</span> <span class="n">cpu_bound</span><span class="p">)</span>
<span class="k">return</span> <span class="n">web</span><span class="p">.</span><span class="n">Response</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="s">"done"</span><span class="p">)</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">web</span><span class="p">.</span><span class="n">Application</span><span class="p">()</span>
<span class="n">app</span><span class="p">.</span><span class="n">add_routes</span><span class="p">([</span><span class="n">web</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'/'</span><span class="p">,</span> <span class="n">hello</span><span class="p">)])</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">web</span><span class="p">.</span><span class="n">run_app</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="threaded-vs-asyncio-results">Threaded vs AsyncIO Results</h2>
<p><img src="/assets/posts/web-app-perf/img/cpu_bound_reqs_sec.png" alt="Asyncio vs Threaded" /></p>
<h3 id="asyncio">AsyncIO</h3>
<p>AsyncIO predictably performs at 2/requests/second. This is because AsyncIO is
single threaded. If your one thread is constantly occupied, it will not be able
to service other requests.</p>
<pre><code class="language-math">1 thread x 0.5 seconds = 2 requests/second
</code></pre>
<h3 id="asyncio-with-thread-executor">AsyncIO with thread executor</h3>
<p>The AsyncIO thread executor variant with results are a less easy to predict but we can try. We are using the default thread pool executor which, from the docs, says:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>will default to the number of processors on the machine, multiplied by 5
</code></pre></div></div>
<p>For me, I have 4 cores on this machine:</p>
<pre><code class="language-math">4 cores x 5 x 0.5 seconds = 40 requests/second
</code></pre>
<p>But wait! In the results, we only obtained around 20 requests/second here. I didn’t
investigate why we didn’t get up to 40 requests/second but can only guess that it
has something to do with python threading just not being very fast and limits to what
4 cores can do in general along with other things running on my computer. Generally, if
you want to maximize python performance, you want to limit the number of CPU-bound
threads per process you are using. This is true for any python application.</p>
<h3 id="threaded-flask">Threaded flask</h3>
<p>Finally, the flask app actually performed worse than I expected with out of the box
documentation for how to run it in production with <code class="language-plaintext highlighter-rouge">gunicorn</code>.</p>
<p>I was surprised to find out <code class="language-plaintext highlighter-rouge">gunicorn</code> defaults to being single-threaded–maybe there
is a lesson in that for performance.</p>
<p>Other than that, flask still performed predictably for the number of threads it
was configured to handle.</p>
<p>It was also interesting to see that AsyncIO with <code class="language-plaintext highlighter-rouge">run_in_executor</code> performed better than <code class="language-plaintext highlighter-rouge">gunicorn</code> with <code class="language-plaintext highlighter-rouge">20</code> threads.</p>
<p>I included results using a different <code class="language-plaintext highlighter-rouge">gunicorn</code> threads setting to be able to match what is happening with the number of threads <code class="language-plaintext highlighter-rouge">run_in_executor</code> is using with AsyncIO.</p>
<h2 id="worst-case">Worst case</h2>
<p>As Mike Bayer mentioned on the Twitter, the worse case scenario with AsyncIO is when
you have a lot of CPU bound code and your application starts performing so poorly
that it can’t even schedule and respond to network IO properly:</p>
<blockquote class="twitter-tweet" data-lang="en"><p lang="en" dir="ltr">no you don't, you don't have database connections being dropped by the DB because the app was hung and couldnt respond to a ping or an authentication challenge. The OS schedules tasks way more effectively than the way your app happens to use IO</p>— mike bayer (@zzzeek) <a href="https://twitter.com/zzzeek/status/1134982063197736962?ref_src=twsrc%5Etfw">June 2, 2019</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>He is correct. If your AsyncIO application has a lot of CPU bound code that is binding
the event loop from operating, you will have problems–potentially very difficult problems to diagnose. These types are issues are less likely to happen with threaded applications because the operating system can schedule running threads regardless of your
application being CPU bound.</p>
<p>Read up on <a href="https://docs.python.org/3/library/asyncio-dev.html">AsyncIO development</a> for tips on understanding how to deal with and avoid this.</p>
<h1 id="network-io-bound">Network IO Bound</h1>
<p>Now where AsyncIO really shines and threaded apps really struggle is with network bound operations.</p>
<p>Most web applications, especially in micro-service frameworks, are network-IO bound. The main point of micro-service frameworks is small services that speak http. If you are blocking when interacting with other services, your performance will dramatically suffer.</p>
<h2 id="setup">Setup</h2>
<p>For the sake of our test, I’ve created a simple slow service that each
application will communicate with to demonstrate the scenario:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">aiohttp</span> <span class="kn">import</span> <span class="n">web</span>
<span class="kn">import</span> <span class="nn">asyncio</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">hello</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mf">0.5</span><span class="p">)</span>
<span class="k">return</span> <span class="n">web</span><span class="p">.</span><span class="n">Response</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="s">'done'</span><span class="p">)</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">web</span><span class="p">.</span><span class="n">Application</span><span class="p">()</span>
<span class="n">app</span><span class="p">.</span><span class="n">add_routes</span><span class="p">([</span><span class="n">web</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'/'</span><span class="p">,</span> <span class="n">hello</span><span class="p">)])</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">web</span><span class="p">.</span><span class="n">run_app</span><span class="p">(</span><span class="n">app</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="mi">8081</span><span class="p">)</span>
</code></pre></div></div>
<p>The service simply does a sleep for half a second and then returns. This
is simulating the half second delay like we did with the CPU bound case.</p>
<p>Our flask app:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="o">@</span><span class="n">app</span><span class="p">.</span><span class="n">route</span><span class="p">(</span><span class="s">'/'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">io_bound</span><span class="p">():</span>
<span class="n">resp</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'http://localhost:8081'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">resp</span><span class="p">.</span><span class="n">text</span>
</code></pre></div></div>
<p>run using <code class="language-plaintext highlighter-rouge">./bin/gunicorn -b localhost:8080 io_bound_flask:app</code>.</p>
<p>And the aiohttp app:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">aiohttp</span> <span class="kn">import</span> <span class="n">web</span>
<span class="kn">import</span> <span class="nn">aiohttp</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">hello</span><span class="p">(</span><span class="n">request</span><span class="p">):</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">app</span><span class="p">.</span><span class="n">session</span>
<span class="k">except</span> <span class="nb">AttributeError</span><span class="p">:</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">app</span><span class="p">.</span><span class="n">session</span> <span class="o">=</span> <span class="n">aiohttp</span><span class="p">.</span><span class="n">ClientSession</span><span class="p">()</span>
<span class="k">async</span> <span class="k">with</span> <span class="n">session</span><span class="p">.</span><span class="n">get</span><span class="p">(</span>
<span class="s">'http://localhost:8081'</span><span class="p">)</span> <span class="k">as</span> <span class="n">resp</span><span class="p">:</span>
<span class="k">return</span> <span class="n">web</span><span class="p">.</span><span class="n">Response</span><span class="p">(</span><span class="n">text</span><span class="o">=</span><span class="k">await</span> <span class="n">resp</span><span class="p">.</span><span class="n">text</span><span class="p">())</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">web</span><span class="p">.</span><span class="n">Application</span><span class="p">()</span>
<span class="n">app</span><span class="p">.</span><span class="n">add_routes</span><span class="p">([</span><span class="n">web</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'/'</span><span class="p">,</span> <span class="n">hello</span><span class="p">)])</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">web</span><span class="p">.</span><span class="n">run_app</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
</code></pre></div></div>
<h2 id="threaded-vs-asyncio-results-1">Threaded vs AsyncIO Results</h2>
<p><img src="/assets/posts/web-app-perf/img/io_bound_reqs_sec.png" alt="Asyncio vs Threaded" /></p>
<h3 id="asyncio-1">AsyncIO</h3>
<p>Here we can see how AsyncIO can perform well when you are connecting to slow
services. The application is single threaded; however, it is not io-bound.</p>
<p>It could perform even better; however, it is simply restricted to the concurrency setting we used for the tests(<code class="language-plaintext highlighter-rouge">100</code>).</p>
<pre><code class="language-math">1 thread x 0.5 seconds x 100 concurrent ~= 200 requests/second
</code></pre>
<h3 id="threaded-flask-1">Threaded flask</h3>
<p>Finally, the flask app performed about the same as the CPU bound example.</p>
<p>Unfortunately, now the flask app is blocking when the CPU isn’t even doing anything.</p>
<h1 id="summary">Summary</h1>
<p>As with most technology, you just need to understand the tradeoffs you’re making and the consequences of them.</p>
<p>You definitely do not want to have CPU bound AsyncIO applications. Also, if the domain logic of your web application is rather CPU-intense(or could be some day), it probably
will start performing badly.</p>
<p>However, thread-based Python web applications can be CPU bound as well and suffer similar performance issues in these scenarios.</p>
<p>In network-io bound scenarios, AsyncIO really shines. If you are communicating with many other services, you might want to look into using AsyncIO.</p>
<h1 id="tips-on-deploying-and-scaling-python-web-applications">Tips on deploying and scaling Python web applications</h1>
<p>Python isn’t fast. That is a tradeoff we all made when we started using it. It can be scaled though and there are a lot of very large sites that run python fine.</p>
<p>This blog post is already too long but here are some parting tips to keep in mind when scaling web applications:</p>
<ul>
<li>load balancers: Run your application with multiple processes and use a load balancer to share traffic between them.</li>
<li>cpu monitoring: Make sure your application isn’t maxed out on CPU. Even better, if you’re using kubernetes(or something similar), setup auto scalers when CPU hits thresholds.</li>
<li>connection pooling: Prevent simultaneous connections to python applications by using proxies/load balancers that will pool requests in front of your application to ensure your application is servicing a healthy number of requests at a time.</li>
<li>message queues: if you have CPU bound tasks, put them in a queue to be processed by another service. If you have a threaded application and have a lot of network-bound io, you might also want to use a message queue for that</li>
<li>micro-services; if you are using AsyncIO, you can leverage micro-services to off-load potentially CPU-bound operations.</li>
<li>caching: the topic is so broad and so many ways to do it…</li>
</ul>It started with a tweetScaling Python Web Applications: AsyncIO vs Threads2019-03-11T02:30:00+00:002019-03-11T02:30:00+00:00https://www.nathanvangheem.com/posts/2019/03/11/scaling-python-web-applications<p>Redirect</p>
<meta http-equiv="refresh" content="0; url=https://www.nathanvangheem.com/posts/2019/06/11/scaling-python-web-applications.html" />
<link rel="canonical" href="https://www.nathanvangheem.com/posts/2019/06/11/scaling-python-web-applications.html" />RedirectBuilding pyrocksdb on ubuntu2019-02-16T01:30:00+00:002019-02-16T01:30:00+00:00https://www.nathanvangheem.com/posts/2019/02/16/building-rocksdb-with-python<p>The latest versions of pyrocksdb does not build on ubuntu.</p>
<p>It seems the library is not compatible with the latest versions of rocksdb.</p>
<p>First, install dependency libraries:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get <span class="nb">install </span>build-essential libsnappy-dev zlib1g-dev libbz2-dev <span class="se">\</span>
libgflags-dev liblz4-dev libzstd-dev git-core <span class="nt">-y</span>
</code></pre></div></div>
<p>Then, checkout a tag of the 4.x line of rocksdb:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone https://github.com/facebook/rocksdb.git <span class="nt">--branch</span><span class="o">=</span>v4.13.5 /usr/src/rocksdb
</code></pre></div></div>
<p>And finally, build the library:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> /usr/src/rocksdb
make shared_lib
make install-shared <span class="nv">INSTALL_PATH</span><span class="o">=</span>/usr
</code></pre></div></div>
<p>Finally, install pyrocksdb with pip:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip <span class="nb">install </span>Cython
pip <span class="nb">install </span>python-rocksdb
</code></pre></div></div>
<p>Here is a full Dockerfile for it::</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM python:3.7.2
RUN apt-get update
RUN apt-get install build-essential libsnappy-dev zlib1g-dev libbz2-dev \
libgflags-dev liblz4-dev libzstd-dev git-core -y
RUN git clone https://github.com/facebook/rocksdb.git --branch=v4.13.5 /usr/src/rocksdb
RUN cd /usr/src/rocksdb && make shared_lib
RUN cd /usr/src/rocksdb && make install-shared INSTALL_PATH=/usr
WORKDIR /usr/src/app
RUN pip install Cython
RUN pip install python-rocksdb
</code></pre></div></div>The latest versions of pyrocksdb does not build on ubuntu.Why Plone Resource Registries Happened2018-09-17T01:30:00+00:002018-09-17T01:30:00+00:00https://www.nathanvangheem.com/posts/2018/09/17/why-plone-resource-registries-happened<p>It all started with the
<a href="https://plone.org/news/old-news/zidanca-sprint-report-js-less-integration">2014 Zidandca Sprint</a>.</p>
<p>We had <a href="https://github.com/plone/mockup">the mockup project</a>. We had <a href="https://plone.org">Plone</a>.
Now, we needed a way to make them work in “harmony” together.</p>
<p>It was the first time I met <a href="https://twitter.com/bloodbare">Ramon Navarro Bosch</a>. Side note: since then,
I’ve done a lot of open source work with Ramon. I even joined one of his startups.</p>
<h2 id="the-mockup-project">The mockup project</h2>
<p>The mockup project was a project which aimed to bring modern JavaScript development to the open source Plone
project. It did this by creating a pure JavaScript project which frontend development was done on separate
from the Plone CMS.</p>
<p>Previously in Plone, JavaScript development was coupled with the CMS. You would register your JavaScript
file to be installed onto the site. Then the CMS would concatenate all the JavaScript and CSS files together
and output a set of files that were included in the html of the page.</p>
<p>The idea of mockup was to decouple frontend development from the CMS and to come up to speed with modern JavaScript.</p>
<h2 id="the-problem">The problem</h2>
<p>Well, decoupling can be great; however, we still need to deliver a CMS. Within the Plone community, it was
understood that you needed to be able to customize and rebuild JavaScript and CSS files within the CMS(I think
now people are not as interested in this feature).</p>
<p>The original author of the mockup project was not interested in merging the two worlds. In fact, if I remember correctly,
I believe his intention was that if people wanted to customize the JavaScript components shipped with Plone,
they should fork the project and make the customizations here.</p>
<p>To give you an idea at the state of JavaScript in 2014, here are the type of components we’re talking about:</p>
<ul>
<li>Grunt</li>
<li>Backbone js</li>
<li>JQuery</li>
<li>RequireJS</li>
<li>LESS CSS</li>
</ul>
<p>(ReactJS was just getting started and we had a few things written in it as well)</p>
<p>What we were asked to solve:</p>
<ul>
<li>integrate mockup within plone</li>
<li>add/remove js/css resources</li>
<li>customize resources through the web</li>
<li>build js/css files through the web</li>
</ul>
<p>Also keep in mind, we were asked to solve these problems mostly for people who still wanted to do
TTW(through the web) development. Ramon and I did not do TTW development and JavaScript technology
does not jive very well with this idea. In other words, we were asked to solve a problem for which
we had no use-case ourselves.</p>
<h2 id="resource-registries">Resource registries</h2>
<p>From those constraints and the current state of things, we came up with the Plone
<a href="https://docs.plone.org/adapt-and-extend/theming/resourceregistry.html">Resource registry</a>.</p>
<p>Basically, what is does is:</p>
<ul>
<li>allow registering “resources” and “bundles”</li>
<li>a resource is a JavaScript or CSS file</li>
<li>a bundle is a build which combines files and uses RequireJS to resolve resource dependencies</li>
<li>also supports compiling LESS CSS</li>
<li>support building resources on the command line from configured customizations/registrations</li>
</ul>
<h2 id="retrospective">Retrospective</h2>
<p>Well, it didn’t work out well. For the most part, the implementation isn’t perfect and
either people didn’t like it or didn’t understand it. Ramon and I spent a insane amounts
of time working out the issues, documenting and training people on it.</p>
<p>“This is a amazing. You integrated JavaScript development with a CMS. No one has ever done that.” said
NO ONE EVER about the project.</p>
<p>Summary of the problems:</p>
<ul>
<li>Complexity: it was hard to describe the JavaScript dependency chain in configuration and then document/train on that</li>
<li>JavaScript technology was still new and buying into a bunch of tech early was a mistake</li>
<li>RequireJS, LESSC, Backbone: now plone is stuck for a while on it</li>
<li>Buggy and difficult to understand when problems occurred</li>
<li>Difficult to change development paradigms when people are used to traditional frontend development</li>
</ul>
<p>One good thing about it is that it encouraged the community into learning and exploring different approaches
to integrating frontend development with Plone.</p>It all started with the 2014 Zidandca Sprint.Update Cloudflare DNS settings with Python2018-07-15T15:30:00+00:002018-07-15T15:30:00+00:00https://www.nathanvangheem.com/posts/2018/07/15/auto-update-cloudflare-dns<h1 id="introduction">Introduction</h1>
<p>I have been slacking on blogging for quite a long time and here is a simple thing I did today
that I can blog about without putting much thought into it :).</p>
<p>Hopefully someone will find this useful.</p>
<h1 id="problem">Problem</h1>
<p>You have a dynamic IP address at home that you want to point a DNS entry at.</p>
<p>This is fine normally; however, regular home internet lines do not have static IPs.</p>
<h1 id="solution">Solution</h1>
<p>Use Cloudflare’s API with a Python script running with a cronjob to make sure your
DNS entry is always updated with the correct IP address.</p>
<h1 id="requirements">Requirements</h1>
<ul>
<li>Computer running on your home network(in my case, a Raspberry Pi Zero)</li>
<li>Python</li>
<li>DNS managed by Cloudflare</li>
<li>DNS record id to update</li>
</ul>
<h1 id="install-requests-library">Install <code class="language-plaintext highlighter-rouge">requests</code> library</h1>
<p>The script we’ll run just needs the <code class="language-plaintext highlighter-rouge">requests</code> libary installed.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo pip3 install requests
</code></pre></div></div>
<h1 id="the-script">The Script</h1>
<p>Save a file with the following contents into the file <code class="language-plaintext highlighter-rouge">/opt/update-dns.py</code>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">requests</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="c1"># XXX Settings you need to update!!!
</span><span class="n">IP_API</span> <span class="o">=</span> <span class="s">'https://api.ipify.org?format=json'</span>
<span class="c1"># Get CF API Key: https://support.cloudflare.com/hc/en-us/articles/200167836-Where-do-I-find-my-Cloudflare-API-key-
</span><span class="n">CF_API_KEY</span> <span class="o">=</span> <span class="s">'REPLACE ME'</span>
<span class="c1"># Your cloudflare email address
</span><span class="n">CF_EMAIL</span> <span class="o">=</span> <span class="s">'REPLACE@ME.COM'</span>
<span class="c1"># Your zone id is located on the main cloudflare domain dashboard
</span><span class="n">ZONE_ID</span> <span class="o">=</span> <span class="s">'REPLACE ME'</span>
<span class="c1"># Run script once without this set and it'll retrive a list of records for you to find the ID to update here
</span><span class="n">RECORD_ID</span> <span class="o">=</span> <span class="s">''</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">RECORD_ID</span><span class="p">:</span>
<span class="n">resp</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span>
<span class="s">'https://api.cloudflare.com/client/v4/zones/{}/dns_records'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">ZONE_ID</span><span class="p">),</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span>
<span class="s">'X-Auth-Key'</span><span class="p">:</span> <span class="n">CF_API_KEY</span><span class="p">,</span>
<span class="s">'X-Auth-Email'</span><span class="p">:</span> <span class="n">CF_EMAIL</span>
<span class="p">})</span>
<span class="k">print</span><span class="p">(</span><span class="n">json</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">resp</span><span class="p">.</span><span class="n">json</span><span class="p">(),</span> <span class="n">indent</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">sort_keys</span><span class="o">=</span><span class="bp">True</span><span class="p">))</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Please find the DNS record ID you would like to update and entry the value into the script'</span><span class="p">)</span>
<span class="n">sys</span><span class="p">.</span><span class="nb">exit</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span>
<span class="n">resp</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">IP_API</span><span class="p">)</span>
<span class="n">ip</span> <span class="o">=</span> <span class="n">resp</span><span class="p">.</span><span class="n">json</span><span class="p">()[</span><span class="s">'ip'</span><span class="p">]</span>
<span class="n">resp</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">put</span><span class="p">(</span>
<span class="s">'https://api.cloudflare.com/client/v4/zones/{}/dns_records/{}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span>
<span class="n">ZONE_ID</span><span class="p">,</span> <span class="n">RECORD_ID</span><span class="p">),</span>
<span class="n">json</span><span class="o">=</span><span class="p">{</span>
<span class="s">'type'</span><span class="p">:</span> <span class="s">'A'</span><span class="p">,</span>
<span class="s">'name'</span><span class="p">:</span> <span class="s">'foo.bar.com'</span><span class="p">,</span>
<span class="s">'content'</span><span class="p">:</span> <span class="n">ip</span><span class="p">,</span>
<span class="s">'proxied'</span><span class="p">:</span> <span class="bp">False</span>
<span class="p">},</span>
<span class="n">headers</span><span class="o">=</span><span class="p">{</span>
<span class="s">'X-Auth-Key'</span><span class="p">:</span> <span class="n">CF_API_KEY</span><span class="p">,</span>
<span class="s">'X-Auth-Email'</span><span class="p">:</span> <span class="n">CF_EMAIL</span>
<span class="p">})</span>
<span class="k">assert</span> <span class="n">resp</span><span class="p">.</span><span class="n">status_code</span> <span class="o">==</span> <span class="mi">200</span>
<span class="k">print</span><span class="p">(</span><span class="s">'updated dns record for {}'</span><span class="p">.</span><span class="nb">format</span><span class="p">(</span><span class="n">ip</span><span class="p">))</span>
</code></pre></div></div>
<p>In the file, you will need to update the variables <code class="language-plaintext highlighter-rouge">CF_API_KEY</code>, <code class="language-plaintext highlighter-rouge">CF_EMAIL</code>, <code class="language-plaintext highlighter-rouge">ZONE_ID</code> and <code class="language-plaintext highlighter-rouge">RECORD_ID</code>.</p>
<p>Test running the script with:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python3 /opt/update-dns.py
</code></pre></div></div>
<p>If it worked, you would receive output, <code class="language-plaintext highlighter-rouge">updated dns record for</code>.</p>
<h1 id="cron">Cron</h1>
<p>Now, to have this running continuously, we can simply use cron.</p>
<p>Edit your cron file.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>crontab <span class="nt">-e</span>
</code></pre></div></div>
<p>Add the following line:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@hourly /usr/bin/python3.5 /opt/update-dns.py
</code></pre></div></div>
<p>And that should be it!</p>IntroductionPersistent RabbitMQ Connections and Python AsyncIO2017-09-27T19:30:00+00:002017-09-27T19:30:00+00:00https://www.nathanvangheem.com/posts/2017/09/27/rabbitmq-python-long-connections<h1 id="using-rabbitmq-with-asyncio">Using RabbitMQ with AsyncIO</h1>
<p>The defacto library to use with RabbitMQ with Python’s AsyncIO is the
<a href="https://aioamqp.readthedocs.io/">aioamqp library</a>.</p>
<h1 id="issues-with-persistent-connections">Issues with persistent connections</h1>
<p>If you have long running processes built around listening to queues and publishing
to queues, you need to make sure your connection to RabbitMQ stays open and
stable.</p>
<h1 id="rabbitmq-and-heartbeat">RabbitMQ and heartbeat</h1>
<p>RabbitMQ has a feature called heartbeat. The setting to tweak this is not quite
what you’d expect. In fact, if you didn’t do a deep read
<a href="https://www.rabbitmq.com/heartbeats.html">into the documentation</a>
on what heartbeat is doing(or if you relied on the docs from libraries), you
might think the setting meant how often a library would send heartbeats to
RabbitMQ</p>
<h2 id="heartbeat-setting-explained">Heartbeat setting explained</h2>
<blockquote>
<p>The heartbeat timeout value defines after what period of time the peer TCP connection
should be considered unreachable (down) by RabbitMQ and client libraries. This value
is negotiated between the client and RabbitMQ server at the time of connection.</p>
</blockquote>
<h2 id="using-heartbeats-properly">Using heartbeats properly</h2>
<p>If you want to do all you can to prevent connections from getting dropped by
RabbitMQ, you’ll want to have a large number here.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kn">import</span> <span class="nn">aioamqp</span>
<span class="n">transport</span><span class="p">,</span> <span class="n">protocol</span> <span class="o">=</span> <span class="k">await</span> <span class="n">aioamqp</span><span class="p">.</span><span class="n">connect</span><span class="p">(</span>
<span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">,</span>
<span class="s">'guest'</span><span class="p">,</span> <span class="s">'guest'</span><span class="p">,</span>
<span class="n">heartbeat</span><span class="o">=</span><span class="mi">800</span>
<span class="p">)</span>
</code></pre></div></div>
<h2 id="when-are-heartbeats-issued">When are heartbeats issued</h2>
<p>RabbitMQ will issue heartbeats to the server about every 2 seconds if there
is activity; however, if there is no activity, no heartbeat frames will be
sent and you could get disconnected.</p>
<p>However, there is a solution. You can manually send out heartbeats in an asyncio
task.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kn">import</span> <span class="nn">aioamqp</span>
<span class="kn">import</span> <span class="nn">asyncio</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">heartbeat</span><span class="p">(</span><span class="n">protocol</span><span class="p">):</span>
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
<span class="k">await</span> <span class="n">asyncio</span><span class="p">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span> <span class="c1"># issue manual heartbeat every 20 seconds
</span> <span class="k">await</span> <span class="n">protocol</span><span class="p">.</span><span class="n">send_heartbeat</span><span class="p">()</span>
<span class="n">transport</span><span class="p">,</span> <span class="n">protocol</span> <span class="o">=</span> <span class="k">await</span> <span class="n">aioamqp</span><span class="p">.</span><span class="n">connect</span><span class="p">(</span>
<span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">,</span>
<span class="s">'guest'</span><span class="p">,</span> <span class="s">'guest'</span><span class="p">,</span>
<span class="n">heartbeat</span><span class="o">=</span><span class="mi">800</span>
<span class="p">)</span>
<span class="n">asyncio</span><span class="p">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">heartbeat</span><span class="p">(</span><span class="n">protocol</span><span class="p">))</span>
</code></pre></div></div>
<h1 id="handling-other-errors">Handling other errors</h1>
<p>Even with the heartbeat feature, you can still experience issues with disconnections
and other errors.</p>
<p><code class="language-plaintext highlighter-rouge">aioamqp</code> has another feature however that allows you to do handle when any kind
of disconnect occurs. It provides a <code class="language-plaintext highlighter-rouge">wait_closed</code> coroutine, which finishes
when the connection to RabbitMQ is done.</p>
<p>So you can setup an asyncio task to wait for this and handle reconnects when it
comes across it.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="kn">import</span> <span class="nn">aioamqp</span>
<span class="kn">import</span> <span class="nn">asyncio</span>
<span class="k">async</span> <span class="k">def</span> <span class="nf">handle_closed</span><span class="p">(</span><span class="n">protocol</span><span class="p">):</span>
<span class="k">await</span> <span class="n">protocol</span><span class="p">.</span><span class="n">wait_closed</span><span class="p">()</span>
<span class="c1"># reconnect logic here...
</span>
<span class="n">transport</span><span class="p">,</span> <span class="n">protocol</span> <span class="o">=</span> <span class="k">await</span> <span class="n">aioamqp</span><span class="p">.</span><span class="n">connect</span><span class="p">(</span>
<span class="n">host</span><span class="p">,</span> <span class="n">port</span><span class="p">,</span>
<span class="s">'guest'</span><span class="p">,</span> <span class="s">'guest'</span><span class="p">,</span>
<span class="n">heartbeat</span><span class="o">=</span><span class="mi">800</span>
<span class="p">)</span>
<span class="n">asyncio</span><span class="p">.</span><span class="n">ensure_future</span><span class="p">(</span><span class="n">handle_closed</span><span class="p">(</span><span class="n">protocol</span><span class="p">))</span>
</code></pre></div></div>Using RabbitMQ with AsyncIOEmbedding Go and groupcache in Python2017-06-03T07:16:00+00:002017-06-03T07:16:00+00:00https://www.nathanvangheem.com/posts/2017/06/03/embedding-golang-in-python-with-groupcache<h1 id="using-go-in-python">Using Go in Python</h1>
<p><code class="language-plaintext highlighter-rouge">Go(golang)</code> is a very fast and efficient compiled programming language. Much like
how you can build Python C-extensions to speed up your python applications, Python
developers also have the option to build Go components that are embedded into their python.</p>
<h1 id="a-simple-demonstration">A Simple Demonstration</h1>
<p><strong>Before you get started, make sure to install golang.</strong></p>
<p>We’ll create a <code class="language-plaintext highlighter-rouge">Go</code> file named <code class="language-plaintext highlighter-rouge">gcode.go</code> that has a function in it that simply
prints a string. Then in Python, we’ll call that function.</p>
<p>Put the following contents into <code class="language-plaintext highlighter-rouge">gocode.go</code>:</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="k">package</span> <span class="n">main</span>
<span class="k">import</span> <span class="p">(</span>
<span class="s">"fmt"</span>
<span class="s">"C"</span>
<span class="p">)</span>
<span class="c">//export say_hi</span>
<span class="k">func</span> <span class="n">say_hi</span><span class="p">(</span><span class="n">txt</span> <span class="o">*</span><span class="n">C</span><span class="o">.</span><span class="n">char</span><span class="p">)</span> <span class="p">{</span>
<span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="n">C</span><span class="o">.</span><span class="n">GoString</span><span class="p">(</span><span class="n">txt</span><span class="p">))</span>
<span class="p">}</span>
<span class="k">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{}</span></code></pre></figure>
<p>The important bit of the Go code is <code class="language-plaintext highlighter-rouge">//export say_hi</code>. This tells the Go compiler
to export the function to be able to be used in the <code class="language-plaintext highlighter-rouge">.so</code> file we’ll build next.</p>
<p>Also, please notice the use of the Go <code class="language-plaintext highlighter-rouge">C</code> library to convert C types to Go types.</p>
<p>So, next we compile it into an <code class="language-plaintext highlighter-rouge">.so</code> file::</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>go build <span class="nt">-buildmode</span><span class="o">=</span>c-shared <span class="nt">-o</span> gcode.so gcode.go</code></pre></figure>
<p>The <code class="language-plaintext highlighter-rouge">-buildmode=c-shared</code> bit is important here as we need to create an
<code class="language-plaintext highlighter-rouge">.so</code>(shared object) file.</p>
<p>Finally, hook it up with python:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">ctypes</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="n">dir_path</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">realpath</span><span class="p">(</span><span class="n">__file__</span><span class="p">))</span>
<span class="n">lib</span> <span class="o">=</span> <span class="n">ctypes</span><span class="p">.</span><span class="n">cdll</span><span class="p">.</span><span class="n">LoadLibrary</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">dir_path</span><span class="p">,</span> <span class="s">'gcode.so'</span><span class="p">))</span>
<span class="n">go_say_hi</span> <span class="o">=</span> <span class="n">lib</span><span class="p">.</span><span class="n">say_hi</span>
<span class="n">go_say_hi</span><span class="p">.</span><span class="n">argtypes</span> <span class="o">=</span> <span class="p">[</span><span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">say_hi</span><span class="p">(</span><span class="n">txt</span><span class="p">):</span>
<span class="k">return</span> <span class="n">go_say_hi</span><span class="p">(</span><span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span><span class="p">(</span><span class="n">txt</span><span class="p">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'utf8'</span><span class="p">)))</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">say_hi</span><span class="p">(</span><span class="s">'Hello!'</span><span class="p">)</span></code></pre></figure>
<p>The Python wiring here relies on ctypes. After getting some of the cruft out of
arranging the function signatures, it’s really pretty easy.</p>
<h1 id="embedding-groupcache-into-python">Embedding groupcache into Python</h1>
<p><a href="https://github.com/golang/groupcache">groupcache</a> is a great caching library
written in Go. It allows Go applications to implement a shared LRU cache and is
very fast.</p>
<p><code class="language-plaintext highlighter-rouge">groupcache</code> has one caveat: It is a read-only cache. You can not modify values once they are in the cache. This means, the only way you can do invalidation is by issuing a new key for a cache value.</p>
<p>At <a href="https://www.onna.com">Onna</a>, we investigated the feasibility of using
groupcache in Python. It ended up being that we couldn’t use it for our use-case;
however, since Onna likes to open source all that we can, we wanted to open
source the project in case anyone was interested in developing the implementation further.</p>
<p>Of course, our implementation was going to be a <code class="language-plaintext highlighter-rouge">guillotina</code> module. You can
find the package on the <a href="https://github.com/guillotinaweb/guillotina_groupcache">guillotinaweb github organization</a>.</p>
<h2 id="install-the-dependencies">Install the dependencies</h2>
<p>Go has it’s own built-in packaging system::</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>go get github.com/golang/groupcache</code></pre></figure>
<h2 id="integration">Integration</h2>
<p>From there, it’s using the same method as described above for exposing Go
functions to export to the <code class="language-plaintext highlighter-rouge">.so</code> and using <code class="language-plaintext highlighter-rouge">ctypes</code> in Python to setup the
function signatures.</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="k">package</span> <span class="n">main</span>
<span class="k">import</span> <span class="p">(</span>
<span class="s">"github.com/golang/groupcache"</span>
<span class="s">"net/http"</span>
<span class="s">"log"</span>
<span class="s">"errors"</span>
<span class="p">)</span>
<span class="k">import</span> <span class="s">"C"</span>
<span class="c">// temporary storage to be able to set data</span>
<span class="k">var</span> <span class="n">Store</span> <span class="o">=</span> <span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">string</span><span class="p">{}</span>
<span class="k">var</span> <span class="n">peers</span> <span class="o">*</span><span class="n">groupcache</span><span class="o">.</span><span class="n">HTTPPool</span> <span class="o">=</span> <span class="no">nil</span>
<span class="k">var</span> <span class="n">cache</span> <span class="o">*</span><span class="n">groupcache</span><span class="o">.</span><span class="n">Group</span> <span class="o">=</span> <span class="no">nil</span>
<span class="k">var</span> <span class="n">srv</span> <span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Server</span> <span class="o">=</span> <span class="no">nil</span>
<span class="c">//export cache_set</span>
<span class="k">func</span> <span class="n">cache_set</span><span class="p">(</span><span class="n">key</span> <span class="o">*</span><span class="n">C</span><span class="o">.</span><span class="n">char</span><span class="p">,</span> <span class="n">value</span> <span class="o">*</span><span class="n">C</span><span class="o">.</span><span class="n">char</span><span class="p">)</span> <span class="o">*</span><span class="n">C</span><span class="o">.</span><span class="n">char</span> <span class="p">{</span>
<span class="c">// groupcache does not have a way to set a value in the cache(pass through cache)</span>
<span class="c">// so we fake it by setting value in global store and then getting the value immediately..</span>
<span class="c">// Ideally, this is switched out with a backend that retrieves the value</span>
<span class="c">// you want to cache</span>
<span class="k">var</span> <span class="n">gkey</span> <span class="o">=</span> <span class="n">C</span><span class="o">.</span><span class="n">GoString</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
<span class="k">var</span> <span class="n">gvalue</span> <span class="o">=</span> <span class="n">C</span><span class="o">.</span><span class="n">GoString</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="n">Store</span><span class="p">[</span><span class="n">gkey</span><span class="p">]</span> <span class="o">=</span> <span class="n">gvalue</span><span class="p">;</span>
<span class="k">var</span> <span class="n">data</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">cache</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="no">nil</span><span class="p">,</span> <span class="n">gkey</span><span class="p">,</span> <span class="n">groupcache</span><span class="o">.</span><span class="n">StringSink</span><span class="p">(</span><span class="o">&</span><span class="n">data</span><span class="p">))</span>
<span class="nb">delete</span><span class="p">(</span><span class="n">Store</span><span class="p">,</span> <span class="n">gkey</span><span class="p">)</span>
<span class="k">return</span> <span class="n">C</span><span class="o">.</span><span class="n">CString</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="p">}</span>
<span class="c">//export cache_get</span>
<span class="k">func</span> <span class="n">cache_get</span><span class="p">(</span><span class="n">key</span> <span class="o">*</span><span class="n">C</span><span class="o">.</span><span class="n">char</span><span class="p">)</span> <span class="o">*</span><span class="n">C</span><span class="o">.</span><span class="n">char</span> <span class="p">{</span>
<span class="k">var</span> <span class="n">data</span> <span class="o">=</span> <span class="s">""</span>
<span class="n">cache</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="no">nil</span><span class="p">,</span> <span class="n">C</span><span class="o">.</span><span class="n">GoString</span><span class="p">(</span><span class="n">key</span><span class="p">),</span> <span class="n">groupcache</span><span class="o">.</span><span class="n">StringSink</span><span class="p">(</span><span class="o">&</span><span class="n">data</span><span class="p">))</span>
<span class="k">return</span> <span class="n">C</span><span class="o">.</span><span class="n">CString</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="p">}</span>
<span class="c">//export setup</span>
<span class="k">func</span> <span class="n">setup</span><span class="p">(</span><span class="n">addr</span> <span class="o">*</span><span class="n">C</span><span class="o">.</span><span class="n">char</span><span class="p">)</span> <span class="p">{</span>
<span class="k">go</span> <span class="k">func</span><span class="p">()</span> <span class="p">{</span>
<span class="n">peers</span> <span class="o">=</span> <span class="n">groupcache</span><span class="o">.</span><span class="n">NewHTTPPool</span><span class="p">(</span><span class="n">C</span><span class="o">.</span><span class="n">GoString</span><span class="p">(</span><span class="n">addr</span><span class="p">))</span>
<span class="n">cache</span> <span class="o">=</span> <span class="n">groupcache</span><span class="o">.</span><span class="n">NewGroup</span><span class="p">(</span><span class="s">"Cache"</span><span class="p">,</span> <span class="m">64</span><span class="o"><<</span><span class="m">20</span><span class="p">,</span> <span class="n">groupcache</span><span class="o">.</span><span class="n">GetterFunc</span><span class="p">(</span>
<span class="k">func</span><span class="p">(</span><span class="n">ctx</span> <span class="n">groupcache</span><span class="o">.</span><span class="n">Context</span><span class="p">,</span> <span class="n">key</span> <span class="kt">string</span><span class="p">,</span> <span class="n">dest</span> <span class="n">groupcache</span><span class="o">.</span><span class="n">Sink</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
<span class="n">v</span><span class="p">,</span> <span class="n">ok</span> <span class="o">:=</span> <span class="n">Store</span><span class="p">[</span><span class="n">key</span><span class="p">]</span>
<span class="k">if</span> <span class="o">!</span><span class="n">ok</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">errors</span><span class="o">.</span><span class="n">New</span><span class="p">(</span><span class="s">"cache key not found"</span><span class="p">)</span>
<span class="p">}</span>
<span class="n">dest</span><span class="o">.</span><span class="n">SetBytes</span><span class="p">([]</span><span class="kt">byte</span><span class="p">(</span><span class="n">v</span><span class="p">))</span>
<span class="k">return</span> <span class="no">nil</span>
<span class="p">}))</span>
<span class="n">srv</span> <span class="o">:=</span> <span class="o">&</span><span class="n">http</span><span class="o">.</span><span class="n">Server</span><span class="p">{</span><span class="n">Addr</span><span class="o">:</span> <span class="n">C</span><span class="o">.</span><span class="n">GoString</span><span class="p">(</span><span class="n">addr</span><span class="p">),</span> <span class="n">Handler</span><span class="o">:</span> <span class="n">http</span><span class="o">.</span><span class="n">HandlerFunc</span><span class="p">(</span><span class="n">peers</span><span class="o">.</span><span class="n">ServeHTTP</span><span class="p">)}</span>
<span class="k">if</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">srv</span><span class="o">.</span><span class="n">ListenAndServe</span><span class="p">();</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
<span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="s">"ListenAndServe: "</span><span class="p">,</span> <span class="n">err</span><span class="p">)</span>
<span class="p">}</span>
<span class="p">}()</span>
<span class="c">// do it in a go routine so we don't block</span>
<span class="n">log</span><span class="o">.</span><span class="n">Print</span><span class="p">(</span><span class="s">"Running cache server node</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span>
<span class="p">}</span>
<span class="c">//export initialized</span>
<span class="k">func</span> <span class="n">initialized</span><span class="p">()</span> <span class="kt">bool</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">cache</span> <span class="o">!=</span> <span class="no">nil</span>
<span class="p">}</span>
<span class="k">func</span> <span class="n">main</span><span class="p">()</span> <span class="p">{}</span></code></pre></figure>
<p>Finally, the Python wiring::</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">ctypes</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="n">dir_path</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">realpath</span><span class="p">(</span><span class="n">__file__</span><span class="p">))</span>
<span class="n">lib</span> <span class="o">=</span> <span class="n">ctypes</span><span class="p">.</span><span class="n">cdll</span><span class="p">.</span><span class="n">LoadLibrary</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">dir_path</span><span class="p">,</span> <span class="s">'gcache.so'</span><span class="p">))</span>
<span class="n">gget</span> <span class="o">=</span> <span class="n">lib</span><span class="p">.</span><span class="n">cache_get</span>
<span class="n">gget</span><span class="p">.</span><span class="n">restype</span> <span class="o">=</span> <span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span>
<span class="n">gget</span><span class="p">.</span><span class="n">argtypes</span> <span class="o">=</span> <span class="p">[</span><span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="n">key</span><span class="p">):</span>
<span class="k">return</span> <span class="n">gget</span><span class="p">(</span><span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span><span class="p">(</span><span class="n">key</span><span class="p">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'utf8'</span><span class="p">)))</span>
<span class="n">gset</span> <span class="o">=</span> <span class="n">lib</span><span class="p">.</span><span class="n">cache_set</span>
<span class="n">gset</span><span class="p">.</span><span class="n">restype</span> <span class="o">=</span> <span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span>
<span class="n">gset</span><span class="p">.</span><span class="n">argtypes</span> <span class="o">=</span> <span class="p">[</span><span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span><span class="p">,</span> <span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">set</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">value</span><span class="p">,</span> <span class="nb">bytes</span><span class="p">):</span>
<span class="n">value</span> <span class="o">=</span> <span class="n">value</span><span class="p">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">gset</span><span class="p">(</span>
<span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span><span class="p">(</span><span class="n">key</span><span class="p">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'utf8'</span><span class="p">)),</span>
<span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span><span class="p">(</span><span class="n">value</span><span class="p">))</span>
<span class="n">gsetup</span> <span class="o">=</span> <span class="n">lib</span><span class="p">.</span><span class="n">setup</span>
<span class="n">gsetup</span><span class="p">.</span><span class="n">argtypes</span> <span class="o">=</span> <span class="p">[</span><span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span><span class="p">]</span>
<span class="k">def</span> <span class="nf">setup</span><span class="p">(</span><span class="n">addr</span><span class="p">):</span>
<span class="n">gsetup</span><span class="p">(</span><span class="n">ctypes</span><span class="p">.</span><span class="n">c_char_p</span><span class="p">(</span><span class="n">addr</span><span class="p">.</span><span class="n">encode</span><span class="p">(</span><span class="s">'utf8'</span><span class="p">)))</span></code></pre></figure>
<h1 id="guillotina-caching">Guillotina caching</h1>
<p>The original goal of this was to provide a <code class="language-plaintext highlighter-rouge">guillotina</code> database shared cache
implementation that was really fast.</p>
<p>We had to abandon it for a
<a href="https://github.com/guillotinaweb/guillotina_rediscache">redis implementation</a>
because dealing the the read-only aspect of the cache came with it’s own
complexity and issues.</p>
<h1 id="final-thoughts">Final thoughts</h1>
<p>Once I understood the initial setup of how to integrate Go with Python,
it was really nice to work with. Go is a great language. I am going to consider
it instead of c-extensions when I need to optimize my Python code from now on.</p>Using Go in PythonManaging Open Source Contributions2017-03-29T07:16:00+00:002017-03-29T07:16:00+00:00https://www.nathanvangheem.com/posts/2017/03/29/managing-open-source<div data-panel="content" data-max-columns="4">
<div class="mosaic-grid-row row">
<div class="mosaic-grid-cell mosaic-width-full mosaic-position-leftmost col-md-12">
</div>
</div>
<div class="mosaic-grid-row row">
<div class="mosaic-grid-cell mosaic-width-full mosaic-position-leftmost col-md-12">
<div class="movable removable mosaic-tile mosaic-IDublinCore-description-tile">
<div class="mosaic-tile-content">
<span id="form-widgets-IDublinCore-description" class="textarea-widget text-field">Tips on managing your open source contributions</span>
</div>
</div>
</div>
</div>
<div class="mosaic-grid-row row">
<div class="mosaic-grid-cell mosaic-width-full mosaic-position-leftmost col-md-12">
<div class="movable removable mosaic-tile mosaic-plone.app.standardtiles.rawhtml-tile">
<div class="mosaic-tile-content">
<p>Today, I noticed a Plone developer asking a question about galleries...</p>
</div>
</div>
<div class="movable removable mosaic-tile mosaic-castle.cms.tweet-tile">
<div class="mosaic-tile-content">
<div class="castle-tile-wrapper"><div style="padding:5px;" class="pat-tweettext" data-pat-tweettext='{"settings": {"cards": "visible"}, "tweetID": "847009381485072384"}'>
</div>
</div>
</div>
</div>
<div class="movable removable mosaic-tile mosaic-IRichText-text-tile">
<div class="mosaic-tile-content">
<p><a href="https://github.com/collective/collective.plonetruegallery" data-linktype="external" data-urltype="/view" data-val="https://github.com/collective/collective.plonetruegallery">collective.plonetruegallery</a> is one of my oldest open source projects. I originally created it when I was an intern at the University of Wisconsin Oshkosh working under Kim Nguyen. It is a product that allows you to plugin different gallery implementations. So there are various collective.ptg.* packages around if you want to use some different gallery implementation.</p>
<p>The last few years I've handed over the reigns to <a href="https://github.com/espenmn" data-linktype="external" data-urltype="/view" data-val="https://github.com/espenmn">Espen</a> who has had far more interest in the project than I've had. I haven't been using the project myself as I responded with...</p>
</div>
</div>
<div class="movable removable mosaic-tile mosaic-castle.cms.tweet-tile">
<div class="mosaic-tile-content">
<div class="castle-tile-wrapper"><div style="padding:5px;" class="pat-tweettext" data-pat-tweettext='{"settings": {"cards": "visible"}, "tweetID": "847050124165627904"}'>
</div>
</div>
</div>
</div>
<div class="movable removable mosaic-tile mosaic-plone.app.standardtiles.rawhtml-tile">
<div class="mosaic-tile-content">
<p>Anyways, that prompted me to talk about managing open source contributions. So let's move on to the meat of this blog post.</p><h2>What happens to your open source projects<br></h2><p>It's difficult for most people who open source and manage many projects to continue maintenance of them over time. Over time, your focus can shift to other problems. You may no longer even use the package you're supposed to be maintaining.</p><h2>Complexity over time</h2><p>Unless managed appropriately, open source projects tend to get more complex and less focused on the original goal of the project. This is what was expressed in the tweet above and I won't dispute the "over-engineered and ugly" comment very far.</p><p>A simple example of how this happens is from another one of my old open source projects <a href="https://github.com/collective/collective.easyslider/" data-linktype="external" data-urltype="/view" data-val="https://github.com/collective/collective.easyslider/" data-mce-href="https://github.com/collective/collective.easyslider/">collective.easyslider</a>. Originally, the slider configuration probably only consisted of 5 options. Today, there are <a href="https://github.com/collective/collective.easyslider/blob/master/collective/easyslider/interfaces.py#L75" data-linktype="external" data-urltype="/view" data-val="https://github.com/collective/collective.easyslider/blob/master/collective/easyslider/interfaces.py#L75" data-mce-href="https://github.com/collective/collective.easyslider/blob/master/collective/easyslider/interfaces.py#L75">25 different configuration options</a> with a couple different ways to use it. Every option, feature, etc adds more code to maintain and adds to the complexity of the project.</p><h3>How does complexity happen?</h3><p>This usually happens because a user wants to implement their corner case and offers to implement the functionality into the core project. This sounds nice; however, it incurs maintenance and complexity costs.</p><p>I've also been paid as a freelancer to implement additional functionality into the open source projects I manage. As a free lancer, to keep the project focussed, concise and easier to maintain over time, consider shipping an extension of the original project that overrides/extends the features. You can even consider delivering fork of the project.</p><h3>Just say no</h3><p>Keep your projects focused and simple. Be okay with saying no to contributions. It's okay for people to fork and provide their own variant that implements their use-case.</p><h2>Do not open source everything</h2><p>Make sure to be choosey when deciding if you should open source something. Even if you do not mean to maintain an open source project for the general public, putting the repo public in your user github area might welcome a issue submission, direct message or email. Over time, this can be more messages, issues than you want to manage.</p><h3>Where/how to host your code</h3><p>If you're having trouble limiting your open source contributions, here are some tips:</p><ul><li>A good README: be up front, provide clear intentions of how the project might be supported</li><li>Community github organization: For communities like Plone, you can contribute code to a <a href="https://github.com/collective" data-linktype="external" data-urltype="/view" data-val="https://github.com/collective" data-mce-href="https://github.com/collective">community organization</a> to help encourage the rest of the community to collaboratively manage the project. You can be up front that you are open sourcing it and with what intentions you have to continue working on it.</li><li>Public github repo(issues disabled): tells consumers you are cool with open sourcing work but not interesting in maintaining a project around it.</li><li>Github gist: another good way to share code without having to maintain a project around it.</li><li>Private repo: if you have no interest in collaborating with an open source community.</li><li>Github organization: if you choose to build a project around a github organization, that usually says you are invested in maintaining a community around the project. Be prepared to manage users, documentation, issues and packages around that project. Examples being <a href="https://github.com/plone" data-linktype="external" data-urltype="/view" data-val="https://github.com/plone" data-mce-href="https://github.com/plone">Plone</a> and <a href="https://github.com/pylons" data-linktype="external" data-urltype="/view" data-val="https://github.com/pylons" data-mce-href="https://github.com/pylons">Pylons</a>.</li></ul><h2>Handing off/walking away</h2><p>An essential part of managing your open source contributions over time is to be able to hand projects off to others. People you hand a project off to should:</p><ul><li>users of the project</li><li>contributors to the project</li><li>interested in keeping the project going</li></ul><p>Looking back at the <a href="https://github.com/collective/collective.plonetruegallery" data-linktype="external" data-urltype="/view" data-val="https://github.com/collective/collective.plonetruegallery" data-mce-href="https://github.com/collective/collective.plonetruegallery">collective.plonetruegallery</a> project, I haven't made a commit to it in almost 2 years. There have still been fixes and various contributions and people still use the package but I've been completely hands off for a long time.</p>
</div>
</div>
</div>
</div>
</div>Tips on managing your open source contributions