Recently, because I mainly use PHP, I discovered the great TMI.php library by Ghostzero and I'm already using it to manage events from my chat, when I encountered a need: how can I send a one-off message (or any IRC-related action)?

The main issue is that you cannot run anything after connect() because it starts a blocking ReactPHP loop and the code under it will be executed only when the loop closes.

For example, take this code:

use GhostZero\Tmi\Client;
use GhostZero\Tmi\ClientOptions;

$client = new Client(new ClientOptions([
    'identity' => [
        'username' => 'ghostzero',
        'password' => 'oauth:...',
    ],
    'channels' => ['ghostzero'],
]));


$client->connect();

echo "I will be executed only when connect() ends";

If we cannot run our code after connecting, we can run it inside an event, right?

The solution

I found out that the WelcomeEvent is fired only once when your user connects to the chat, so it's the perfect place where to place our logic. We can write our one-off methods and then close the connection:

use GhostZero\Tmi\Events\Irc\WelcomeEvent;

$client
    ->on(WelcomeEvent::class, function () use ($client) {
        $client->say('ghostzero', 'Hello from my code');

        $client->close();
    })
    ->connect();

If you would run this code straight away the script would never close and the WelcomeEvent would trigger more than once. This happens because the package is smart enough to reconnect when the connection to the IRC channel is closed and, guess what, we're creating a beautiful loop! We send a message on Welcome -> we close the connection -> the package reconnects -> a Welcome event is triggered -> we send a message on Welcome -> ... and so on.

We can easily solve this by disabling the auto reconnection in our client settings:

$client = new Client(new ClientOptions([
    'connection' => [
        'reconnect' => false,
    ],
    'identity' => [
        'username' => 'ghostzero',
        'password' => 'oauth:...',
    ],
    'channels' => ['ghostzero'],
]));

If we now run our code it will... do nothing. It will But everything is correct now, doesn't it? It even printed "I will be executed only when connect() ends" !. Well, theoretically it does, but technically it does not for one simple reason: we close the loop before our code is fullfilled. To get it better, IRC methods like say() are not sync, that means that their execution does not mean that the desired action is already done. In our case, $client->close() disconnects the client and stops the loop before say() actually sends something.

The way I found that can solve this is to close the loop after a few seconds, just to be sure that our IRC message is actually sent. We can use the addTimer method of ReactPHP to achieve it:

$client
    ->on(WelcomeEvent::class, function () use ($client) {
        $client->say('ghostzero', 'Hello from my code');

        $client->getLoop()->addTimer(3, fn ()  => $client->close());
    })
    ->connect();

Putting everything back together we will get the following snippet. The loop runs, sends a message in our chat and then ends, showing the message I will be executed only when connect() ends.

use GhostZero\Tmi\Client;
use GhostZero\Tmi\ClientOptions;

$client = new Client(new ClientOptions([
    'connection' => [
        'reconnect' => false,
    ],
    'identity' => [
        'username' => 'ghostzero',
        'password' => 'oauth:...',
    ],
    'channels' => ['ghostzero'],
]));

$client
    ->on(WelcomeEvent::class, function () use ($client) {
        $client->say('ghostzero', 'Hello from my code');

        $client->getLoop()->addTimer(3, fn ()  => $client->close());
    })
    ->connect();

echo "I will be executed only when connect() ends";

Hopefully, in the future, the package will provide a shortcut to avoid this hacky way. Remember to leave a star to the TMI.php project!

We recently moved a big part of my company's core APIs from a Node-Frankenstein to a well-tested Laravel 8 project, but of course, the performance - although opcache is enabled, is not really close to a Node app; but we already knew this.

So... what's next? "Let's try the preloading", I said! It should be easy, right? Weeell...

Approach 1: vendor-centric

I had no idea where to start. The needed result was simple: loop over "desired" files and require_once - or opcache_compile_file - them.

I found time ago the great article "Preloading in PHP7.4" on stitcher by Brent. It also has a full code! That's a dream, omg. Like a child with the Hogwarts Lego I quickly copied and pasted it. Of course, I set the needed configurations: opcache.preload=/app/preload.php and opcache.preload_user=www-data - this last one is needed if your account is root.

Then I restarted my container. And it crashed. Let's see the errors...

Uncaught Error: Class 'Illuminate\Queue\Queue' not found in /app/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php:13

Well, that's fine, I'm just gonna ignore that file (there's a method in the example class to ignore specific files). Time to restart. New error: Uncaught Error: Class 'Opis\Closure\SerializableClosure' not found in /app/vendor/laravel/framework/src/Illuminate/Queue/SerializableClosure.php:7. Ok...? Let's ignore that too. Then another one. And another one.

In the end, I ignored the whole Laravel framework and 0 classes were preloaded. What am I doing wrong? Why does it work in the article? I'll never know this, but let's try to replace require_once in the class with opcache_compile_file.

Now restart and... It started! But there are a few errors like Can't preload unlinked class Illuminate\Database\PDO\PostgresDriver: Unknown parent Doctrine\DBAL\Driver\AbstractPostgreSQLDriver in /app/vendor/laravel/framework/src/Illuminate/Database/PDO/PostgresDriver.php on line 8, I guess I can filter out those files later and we're good.

Time to benchmark it! First, let's test the performance without preload. Note that the project is running in a Docker container with opcache enabled, XDebug and other 20+ containers alive. And if you're asking yourself "What's ab?", the answer is: Apache Bench.

    ab -n 100 -c 5 -l http://mysite.localhost/page

    Requests per second:    21.78 [#/sec] (mean)
    Time per request:       229.516 [ms] (mean)
    Time per request:       45.903 [ms] (mean, across all concurrent requests)

That's our status quo. Now let's see with preload.

Requests per second:    25.03 [#/sec] (mean)
Time per request:       199.774 [ms] (mean)
Time per request:       39.955 [ms] (mean, across all concurrent requests)

That's an increase for sure! We gained ±14.9% req/s and ±12.9% time/req!

Can it be better? Giving a shot to the logs, it preloaded 1022 classes and, in the list of preloaded files, there are a lot of dev-dependencies (like Telescope, Tinker, Testing) and some unusued classes too (Session, Console...). Let's strip them out and see if something changes. I'm just gonna put the namespaces in the ignore() method like this:

(new Preloader())
    ->paths(__DIR__ . '/vendor/laravel')
    ->ignore(
        'Laravel\Telescope',
        'Laravel\Tinker',
        'Illuminate\Queue',
        'Illuminate\Contracts\Queue',
        'Illuminate\View',
        'Illuminate\Contracts\View',
        'Illuminate\Foundation\Console',
        'Illuminate\Notification',
        'Illuminate\Contracts\Notifications',
        'Illuminate\Bus',
        'Illuminate\Session',
        'Illuminate\Contracts\Session',
        'Illuminate\Console',
        'Illuminate\Testing',
        'Illuminate\Http\Testing',
        'Illuminate\Support\Testing',
        'Illuminate\Cookie',
        'Illuminate\Contracts\Cookie',
        'Illuminate\Broadcasting',
        'Illuminate\Contracts\Broadcasting',
        'Illuminate\Mail',
        'Illuminate\Contracts\Mail',
    )
    ->load();

Did something change? Restart again, now there are 625 preloaded classes, run ab and the results:

Requests per second:    25.22 [#/sec] (mean)
Time per request:       198.274 [ms] (mean)
Time per request:       39.655 [ms] (mean, across all concurrent requests)

That's pretty much the same, nevermind, but I want to make a final test: add more vendors to paths() method:

(new Preloader())
      ->paths(
        __DIR__ . '/vendor/psr',
        __DIR__ . '/vendor/monolog',
        __DIR__ . '/vendor/doctrine',
        __DIR__ . '/vendor/guzzlehttp',
        __DIR__ . '/vendor/ramsey/uuid',
        __DIR__ . '/vendor/ramsey/collection',
        __DIR__ . '/vendor/vlucas/phpdotenv',
        __DIR__ . '/vendor/symfony',
        __DIR__ . '/vendor/laravel',
        __DIR__ . '/vendor/nesbot/carbon',
        __DIR__ . '/vendor/sentry',
        __DIR__ . '/vendor/auth0',
      )

And running again the ab gives us:

Requests per second:    26.54 [#/sec] (mean)
Time per request:       188.400 [ms] (mean)
Time per request:       37.680 [ms] (mean, across all concurrent requests)

Ok, that's something, right? And that "something" is ±21.8% req/s and ±17.9% time/req from the no-preload checks.

Now let's group and compare all the data:

Approach 2: "Hot" files

The second approach I tried was to identify the hot files using Laraload, a wrapper for Laravel of Preloader; it creates a global middleware for your app and for every 500 requests (or your custom logic) it generates a preload.php with all the detected files. Pretty straightforward.

I "forced" the requests using ApacheBench with 500 requests and in the preload.php now there are 1490 classes. Wow, the problem is that they include dev-dependencies too like Telescope or Tinker, so I started removing them - also because they would not be found in production and I went down to 1351 classes of both vendors and files of my app.

Like in the Approach 1 there were some errors, for example, I had to remove the local app/Http/Kernel.php from the preloaded files and finally, the PHP container started again.

Remember our no-preload status:

ab -n 100 -c 5 -l http://mysite.localhost/page

Requests per second:    21.78 [#/sec] (mean)
Time per request:       229.516 [ms] (mean)
Time per request:       45.903 [ms] (mean, across all concurrent requests)

As usual, I ran my Apache Bench command to check the performance aaaand my app crashed because of amphp - I don't even know which vendor uses it but nevermind -, so I removed all its references from the big files array, restarted again and ran the bench command.

Requests per second:    25.16 [#/sec] (mean)
Time per request:       198.694 [ms] (mean)
Time per request:       39.739 [ms] (mean, across all concurrent requests)

Doing the math we gained ±15.5% req/s and ±13.4% time/request, but I wanted to make another test: what happens if I keep only Laravel vendor in the array? This is the result:

Requests per second:    25.03 [#/sec] (mean)
Time per request:       199.787 [ms] (mean)
Time per request:       39.957 [ms] (mean, across all concurrent requests)

Well, it's pretty much the same.

Final thoughts

Ok kids, time to bring something to the table. All the data to me!

As we can see all the preload methods produce a similar performance boost, but one thing is sure: you should definitely preload.

How to preload is up to you, I will probably stick to the Preloader class from Brent (Sticher.io) because it's easier to read and maintain.

Staging benchmarks

The real test: how does this impact on a staging environment? I released this for our company in our staging environment using the "Refined preload + vendors" method. Our APIs are running on K8S and I took some before-after benchmarks. Below you can see the results; doing our math, we gained ±10.5% req/s and ±9.5% time/request.

Before preload:

Requests per second:    12.22 [#/sec] (mean)
Time per request:       409.068 [ms] (mean)
Time per request:       81.814 [ms] (mean, across all concurrent requests)

After preload:

Requests per second:    13.51 [#/sec] (mean)
Time per request:       370.110 [ms] (mean)
Time per request:       74.022 [ms] (mean, across all concurrent requests)

Summary:

Will be updated soon with Production benchmarks!

Why Bash and not another language? Since I had to use curl -I I thought it was simpler in Bash and it is more portable; let's see the result.

The first thing is to loop line by line a file, which will contain our URLs and I found on StackOverflow this while loop to achieve it. It will take the filename from the args when you run the script (done < "$1").

#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
  echo "[${i}] Doing $line"
done < "$1"

At this time it just prints each line, so we need to capture the output of curl to get the HTTP Code.

#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
  echo "[${i}] Doing $line"
  statusCode=$(curl -s -o /dev/null -I -w "%{http_code}" $line)
done < "$1"

Let's explain a bit of the curl arguments:

• -s hides the progress of the "call";

• -o /dev/null hides the body of the results;

• -I fetches the headers only (HEAD method);

• -w "%{http_code}" writes the HTTP Status Code to stdout after exit.

Now the most important check: is the status code a valid one? Right after the declaration of statusCode, let's add this:

if [[ "$statusCode" != 2* ]]; then
    echo "Code: $statusCode"
fi

We are telling the script: if the content of statusCode doesn't start with "2" (200, 201, 204 and so on), print the error.

The script is finished and its final results will be this:

#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
  echo "[${i}] Doing $line"
  statusCode=$(curl -s -o /dev/null -I -w "%{http_code}" $line)

  if [[ "$statusCode" != 2* ]]; then
    echo "Code: $statusCode"
  fi
done < "$1"

Let's try it. Create a file called urls.txt with this content and save it in the same directory of our script (status_code.sh):

http://google.com
https://twitter.com/icantfindafakeuser
https://www.facebook.com

Now run it with bash status_code.sh urls.txt and the output will look like this:

Improvements

The script works pretty well, but if you have a lot of URLs it could be hard to retrieve all the errors, so we can store them in an array and print them at the end of the script.

We will add an empty array at the beginning of the file and then push the "wrong" URL to it when it doesn't return a 2xx status code.

#!/bin/bash
# Empty array
errors=()

while IFS='' read -r line || [[ -n "$line" ]]; do
  echo "Doing $line"
  statusCode=$(curl -s -o /dev/null -I -w "%{http_code}" $line)
  if [[ "$statusCode" != 2* ]]; then
    echo "Code: $statusCode"
    errors+=("[${statusCode}] ${line}")
  fi

  sleep 2
done < "$1"

Now, after the while loop, we are going to print the number of errors and their content.

Note: I have added a sleep 2 to create a delay of 2 seconds between each curl.

#!/bin/bash
# Empty array
errors=()

while IFS='' read -r line || [[ -n "$line" ]]; do
  echo "Doing $line"
  statusCode=$(curl -s -o /dev/null -I -w "%{http_code}" $line)
  if [[ "$statusCode" != 2* ]]; then
    echo "Code: $statusCode"
    errors+=("[${statusCode}] ${line}")
  fi

  sleep 2
done < "$1"

echo "---------------"
errorsCount=${#errors[@]}
echo "Found $errorsCount errors."

if (( $errorsCount > 0 )); then
  printf '%s\n' "${errors[@]}"
fi

The printf in the last condition will print the items in our array one-per-line, otherwise, they would be all in the same line, hard to read.

If we run again our script, always with bash status_code.sh urls.txt, the new output will be the following:

Of course, it there won't be errors, the array won't be printed.

The final touch: line offset

The last improvement we can make is to add the possibility to choose the offset of the script, so we can specify from which line we want to start. This is useful if we have a lot of URLs and our script dies or there is some unexpected error.

We will create a i variable incremented in each iteration of the loop. If, when we run the script, there is a second (actually third, the index starts from 0) argument ("$2"), we will check if our i is less than it and, in case, we will skip the current URL.

#!/bin/bash
# Empty array
errors=()
i=0

while IFS='' read -r line || [[ -n "$line" ]]; do
  # Increment the variable  
  ((i++))

  # Skip if offset if specified and current index less than it
  if [[ "$2" && $i -lt "$2" ]]; then
    continue
  fi

  echo "[${i}] Doing $line"
  statusCode=$(curl -s -o /dev/null -I -w "%{http_code}" $line)
  if [[ "$statusCode" != 2* ]]; then
    echo "Code: $statusCode"
    errors+=("[${statusCode}] ${line}")
  fi

  sleep 2
done < "$1"

echo "---------------"
errorsCount=${#errors[@]}
echo "Found $errorsCount errors."

if (( $errorsCount > 0 )); then
  printf '%s\n' "${errors[@]}"
fi

We added also a "tracker" in the line printing to know the current line index. Let's try it out: for example, we want to start from the second line: bash status_code.sh urls.txt 2.

This is it, hoping it helped.

Some days ago, I had the need to upload multiple files, but one per (at least) second, because on the server side, I put a timestamp in seconds near the image and it would be the same for every item uploaded. Since I couldn't switch to milliseconds' timestamps and Dropzone doesn't have a built-in function to create a delay between uploads, I found this solution, thanks also to some Stackoverflow answers.

Note that this code won't work if you use autoProcessQueue, because we don't know when users finished choosing its files, so we init the uploads from a button.

Basic HTML

In our HTML we need to include the CSS (https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.css) and the JS (https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.js).

<!DOCTYPE html>
<html>
<head>
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.css" />
    <title>Our magic Dropzone</title>
</head>
<body>

    <script src="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.js"></script>
</body>
</html>

Now, let's create a form where Dropzone will be available.

<!DOCTYPE html>
<html>
<head>
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.css" />
    <title>Our magic Dropzone</title>
</head>
<body>
    <form action="/file-upload" class="dropzone" id="my-awesome-dropzone"></form>

    <button id="submit-files">Upload</button>

    <script src="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.js"></script>
</body>
</html>

Since Dropzone has autodiscovery enabled by default, it will be immediately working; we are going to disable this to create our DZ instance to use its method. We also added a button that will trigger the upload of the files.

JavaScript

We need to adjust the settings of our Dropzone form and initialize it programmatically, disabling the autoDiscover functionality and disabling the auto queue, otherwise selected files are automatically uploaded.

Dropzone.autoDiscover = false;
const myDropzone = new Dropzone('#my-awesome-dropzone', {
  autoProcessQueue: false,
});

Now it's time to connect the button: when clicked, it will fetch all the files inside our form and, with the processFile method of DZ, we'll tell to Dropzone which file to process.

const $button = document.getElementById('submit-files');

$button.addEventListener('click', function () {
  // Retrieve selected files
  const acceptedFiles = myDropzone.getAcceptedFiles();
  for (const i = 0; i < acceptedFiles.length; i++) {
    myDropzone.processFile(acceptedFiles[i]);
  }
});

If you try to click the button, the files will be uploaded, but it works like the default Dropzone, so are going to add a timeout inside the for loop, where the milliseconds of timeout will be the current index * our desired delay, for example, 2000 (2 seconds).

const $button = document.getElementById('submit-files');

$button.addEventListener('click', function () {
  // Retrieve selected files
  const acceptedFiles = myDropzone.getAcceptedFiles();
  for (consti = 0; i < acceptedFiles.length; i++) {
    setTimeout(function () {
      myDropzone.processFile(acceptedFiles[i]);
    }, i * 2000);
  }
})

What will happen now? The first file will be processed instantly, because its index is 0 and 0x2000 = 0; the second file will be processed after 2s, the third after 4s from the first file, which means it will be uploaded after 2s from the second file.

Example

To check if it correctly works, we can log in the console the current time in the timeout. This would be the output (the upload endpoint doesn't exists in my case):

Final code and working demo

The final code, putting it all together, will be this.

<!DOCTYPE html>
<html>
<head>
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.css" />
    <title>Our magic Dropzone</title>
</head>
<body>
    <form action="/file-upload" class="dropzone" id="my-awesome-dropzone"></form>

    <button id="submit-files">Upload</button>

    <script src="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.js"></script>
    <script>
    // Init dropzone instance
    Dropzone.autoDiscover = false;
    const myDropzone = new Dropzone('#my-awesome-dropzone', {
      autoProcessQueue: false,
    });

    // Submit
    const $button = document.getElementById('submit-files');
    $button.addEventListener('click', function () {
      // Retrieve selected files
      const acceptedFiles = myDropzone.getAcceptedFiles();
      for (const i = 0; i < acceptedFiles.length; i++) {
        setTimeout(function () {
          myDropzone.processFile(acceptedFiles[i]);
        }, i * 2000);
      }
    })
    </script>
</body>
</html>

You can see a working demo within this CodePen or here below.

In the most recent days, the Instagram team decided (probably after Cambridge Analytica?) to restrict their APIs, so the parameter is not more available, returning a 403 Forbidden error.

So, what to do now? Should I create an app on Instagram? Access tokens? OAuth?

Nothing like that. The JSON of Instagram is always in the public profile of users, so we just need to get it with a simple RegExp.

Note: this works, of course, only for public profiles.

In the source of a profile, we can find the window._sharedData variable which is a big object with a lot of links; those links are posts. The HTML is this:

<script type="text/javascript">window._sharedData = { ... };</script>

With a simple RegExp, we will be able to fetch its content and convert it into a readable object from our code.

Code

In this case, we'll use Axios as the HTTP library and Node.js as language (JavaScript) with async/await feature.

async function instagramPhotos() {
    const userInfoSource = await Axios.get('https://www.instagram.com/theraloss/');
}

Now that we have the source, we need to write the RegExp. Since window._sharedData is an object, we can then read it with JSON.parse function.

async function instagramPhotos() {
    const userInfoSource = await Axios.get('https://www.instagram.com/theraloss/');

    // userInfoSource.data contains the HTML from Axios
    const jsonObject = userInfoSource.data.match(/<script type="text\/javascript">window\._sharedData = (.*)<\/script>/)[1].slice(0, -1)
}

Here we do three things:

1. We use Foo as RegExp to get all the JSON;

2. We retrieve the 2° item ([1]) of the returned matches from .match() JS function, since the 1° is the whole contains also our "delimitator";

3. We delete the last character of our object because, on the source page, it ends with ; and, of course, it won't be a valid JSON.

Now that we have our JSON, we can easily read it.

async function instagramPhotos() {
    const userInfoSource = await Axios.get('https://www.instagram.com/theraloss/');

    // userInfoSource.data contains the HTML from Axios
    const jsonObject = userInfoSource.data.match(/<script type="text\/javascript">window\._sharedData = (.*)<\/script>/)[1].slice(0, -1);

    return JSON.parse(jsonObject);
}

That's it! Now you can access the most recents posts with the path userInfo.entry_data.ProfilePage[0].graphql.user.edge_owner_to_timeline_media.edges. If, for example, we want to retrieve the most recent 10 images - excluding the videos - and store them in an array, we could write a code like this.

async function instagramPhotos() {
    // It will contain our photos' links
    const res = [];

    try {
        const userInfoSource = await Axios.get('https://www.instagram.com/theraloss/');

        // userInfoSource.data contains the HTML from Axios
        const jsonObject = userInfoSource.data.match(/<script type="text\/javascript">window\._sharedData = (.*)<\/script>/)[1].slice(0, -1);

        const userInfo = JSON.parse(jsonObject);
        // Retrieve only the first 10 results
        const mediaArray = userInfo.entry_data.ProfilePage[0].graphql.user.edge_owner_to_timeline_media.edges.splice(0, 10);
        for (let media of mediaArray) {
            const node = media.node;

            // Process only if is an image
            if ((node.__typename && node.__typename !== 'GraphImage')) {
                continue;
            }

            // Push the thumbnail src in the array
            res.push(node.thumbnail_src);
        }
    } catch (e) {
        console.error('Unable to retrieve photos. Reason: ' + e.toString());
    }

    return res;
}

In the era of the conversion rate, where a user sign-up is fundamental and must be quick, the social login became a required tile of the UX. Not having this option will probably reduce the registrations, but nowadays enabling this feature is really simple, since every programming language has a lot of libraries that, with few lines of code, create a complete oAuth flow.

In this article we'll explore the case of Go with Gocialite, a Socialite (Laravel) inspired package very simple to use.

The idea of Gocialite is to provide both an easy package but also a hackable one. I was trying Goth, but with Revel, it was very hard to make it working, so why don't write a library more simple? For example, if you want to contribute, you just have to clone a file, change same variable et voilà.

But now, let's coding.

For this example we'll use Gin as router, but you can use as wall whatever you want, like Gorilla/Mux or others.

Starting code and homepage

The first thing is of course to create a new Go project, in this case we'll name it "codelikepro/social". Inside it, there will be only a file named main.go. We'll use terminal, but of course you can create folders and the file manually.

$ mkdir -p $GOPATH/src/github.com/codelikepro/social
$ touch $GOPATH/src/github.com/codelikepro/social/main.go

Now that our file exists, we need to start writing the package name and the necessary imports.

First of all, install gocialite and gin with go get:

$ go get github.com/danilopolani/gocialite
$ go get github.com/gin-gonic/gin

These are the only two external packages we'll use, so we are ready to code. Note I commented fmt and net/http packages because we'll need them later.

Open main.go with our favorite editor and write the following:

package main

import (
    // "fmt"
    // "net/http"

    "github.com/danilopolani/gocialite"
    "github.com/gin-gonic/gin"
)

// Define our gocialite instance
var gocial = gocialite.NewDispatcher()

func main() {

}

At this point in our main function we'll initialize our Gin router and render a homepage with a simple button Sign in with Github.

func main() {
    router := gin.Default()

    router.GET("/", indexHandler)
    // router.GET("/auth/github", redirectHandler)
    // router.GET("/auth/github/callback", callbackHandler)

    router.Run("127.0.0.1:9090")
}

// Show the homepage
func indexHandler(c *gin.Context) {
    c.Writer.Write([]byte("<html><head><title>Gocialite example</title></head><body>" +
    "<a href='/auth/github'><button>Login with GitHub</button></a>" +
    "</body></html>"))
}

// Redirect user to social login
func redirectHandler() {

}

// Collect user info
func callbackHandler() {

}

With Gin we declared three routes:

• A homepage, responding to route /

• A redirect bridge at /auth/github. This page will only redirect the user to correct login of the provided driver.

• A callback page at /auth/github/callback where we'll collect the user information.

If we test our script running, inside our project folder, go run main.go, we'll just see a button – not working, obviously.

Redirect the user

The next step is to create our bridge, where we retrieve the redirect URL and we send the user to it.

Uncomment the route router.GET("/auth/github", redirectHandler) and let's write the redirectHandler function.

If you don't have a GitHub app, go to New OAuth Application and create a new one. Make sure to use http://localhost:9090/auth/github/callback as Authorization callback URL.

In the next step you'll Client ID and Client Secret, to populate our appSettings.

With our client information obtained, we are ready to write the code to redirect the user.

// Redirect user to social login
func redirectHandler(c *gin.Context) {
    // Define our settings
    appSettings := map[string]string{
        "clientID":     "xxxxxxxxxxxxxx",
        "clientSecret": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "redirectURL":  "http://localhost:9090/auth/github/callback",
    }

    // Retrieve the URL
    authURL, err := gocial.New().
        Driver("github").
        Scopes([]string{"public_repo"}). // Specify custom scopes
        Redirect(
            appSettings["clientID"],
            appSettings["clientSecret"],
            appSettings["redirectURL"],
        )

    // Check for errors
    if err != nil {
        c.Writer.Write([]byte("Error: " + err.Error()))
        return
    }

    // Redirect to authURL with status 302
    c.Redirect(http.StatusFound, authURL)
}

We used Driver() method to specify our provider, in this case, Github, but you can use a lot of other social. Then, with Scopes(), we defined our custom scopes. In our case, we want access to its public repositories, but if you don't need custom privileges you can omit the function. Note that we didn't pass "user" scopes because Gocialite includes them by itself.

Finally, in the Redirect() function we pass our information and we retrieve our authURL.

Before redirecting the user, we check for errors and then, with c.Redirect(), a Gin function, we send the user to GitHub login approval with status 302 (Temporary redirect) taken from the net/http package (click it to see all the available status). Remember to uncomment the "net/http" line in our imports!

Get user info

The last part of our flow is the callback to handle user information and we are going to write it in the callbackHandler function.

Here we have to pass to the Handle() method two values retrievable from the query string: state and code. State is a secret string that Gocialite passed in the beginning to the provider and now it checks if it's equal, in order to block CSRF attacks. Code is a string needed to start the exchange with the provider for user authentication.

It will return three variables: a user, a token and an error. If there's an error, the first two will be nil, otherwise the error will be nil.

func callbackHandler(c *gin.Context) {
    // Retrieve query params for state and code
    state := c.Query("state")
    code := c.Query("code")

    // Handle callback and check for errors
    user, token, err := gocial.Handle(state, code)
    if err != nil {
        c.Writer.Write([]byte("Error: " + err.Error()))
        return
    }

    // Print token information
    fmt.Printf("%#v", token)
    // Print in terminal user information
    fmt.Printf("%#v", user)

    // If no errors, show provider name
    c.Writer.Write([]byte("Hi, " + user.FullName))
}

If everything went well, it will print in your browser "Hi, NAME", in my case "Hi, Grork".

We also use fmt.Printf with the %#v format to print in the terminal the full user and token content, but in production you should remove them.

The user variable will contain the main information of the current logged-in user:

• ID

• Username

• FirstName

• LastName

• FullName

• Email

• Avatar

• Raw

They will be all strings except for raw, which is a map of interfaces (map[string]interface{}) containing the original JSON returned by the provider. Note that all fields (except Raw) could be empty.

The token will be an oauth2.Token struct containing the following data:

• AccessToken string

• TokenType string

• RefreshToken string

• Expiry time.Time

If you want to set up a multi provider social login, here there is a full example.

If you like Gocialite, feel free to leave a star!

The crawler will follow this flow:

1. At the very beginning, it retrieves a list of valid HTTPS proxies from sslproxies.org

1. At the very beginning, it retrieves a list of valid HTTPS proxies from sslproxies.org

   1. With BeautifulSoup, it scans the table and obtains the values

   2. 1.2. With the Python module Fake-useragent, simulates a user agent to simulate a real request

2. It does its stuff and calls (in this case, we use a range from 1 to 100 to test it)

3. Every 10 requests (you can of course increment this value), he changes proxy

4. If the proxy is broken, it deletes it from our list and catches another one.

Prerequisites

Our code will be compatible with Python 3.x, so make sure you have it installed. We are going to use also two external modules:

• BeautifulSoup, to scrape the source. Install it with pip3 install beautifulsoup4

• Fake-useragent to retrieve a random User Agent, to simulate a "real" request. Install it with pip3 install fake-useragent. Note: This is optional. You can replace it with a "hard coded" and fixed string, it doesn't matter.

Coding a hot damn script

The first part of every language is the "inclusion" section, so we are going to include our needed libraries.

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
from fake_useragent import UserAgent
import random

Now we generate a random user agent and initialize an empty list where we'll put our proxies.

ua = UserAgent() # From here we generate a random user agent
proxies = [] # Will contain proxies [ip, port]

Since we like a structure where the most important code is at the top and the other small functions at the bottom (and in Python this is "not possible"), our code will be in a method called main which will be recalled at the end, like this:

# Main function
def main():
    # Our code here

if __name__ == '__main__':
  main()

Retrieve proxies

Because we want a rotating proxy system, we need to retrieve them from the beginning and we'll achieve that by scraping the site sslproxies.org with BeautifulSoup (BS from now), the most famous scraping library.

Inside our main function, we initialize a new Request, attach a User Agent to it and retrieve its contents.

# Retrieve latest proxies
proxies_req = Request('https://www.sslproxies.org/')
proxies_req.add_header('User-Agent', ua.random)
proxies_doc = urlopen(proxies_req).read().decode('utf8')

Now creating a new BS instance, we scan all the rows contained in the main table.

soup = BeautifulSoup(proxies_doc, 'html.parser')
proxies_table = soup.find(id='proxylisttable')

# Save proxies in the array
for row in proxies_table.tbody.find_all('tr'):
  proxies.append({
    'ip':   row.find_all('td')[0].string,
    'port': row.find_all('td')[1].string
  })

With proxies.append, we save a new dict with the keys ip and port. At this point, all we have to do is to get a random proxy from our list. Since we need this many times, we'll put the code inside a function called random_proxy, right before the line if __name__ == '__main__':

# Retrieve a random index proxy (we need the index to delete it if not working)
def random_proxy():
  return random.randint(0, len(proxies) - 1)

Note that we return the key and not the value of the proxy: why this? Because if a proxy fails, we want to remove it from the list, so we won't catch it again.

Request, request, request

For testing purposes, we'll make 100 requests to icanhazip.com which will return our current IP (proxied, of course). Using the random_proxy method written above, we retrieve a proxy and then attach it to our request and every 10 requests we change IP.

proxy_index = random_proxy()
proxy = proxies[proxy_index]

for n in range(1, 100):
    req = Request('http://icanhazip.com')
    req.set_proxy(proxy['ip'] + ':' + proxy['port'], 'http')

    # Every 10 requests, generate a new proxy
    if n % 10 == 0:
      proxy_index = random_proxy()
      proxy = proxies[proxy_index]

Now with a try/catch we intercept broken proxies and delete them from our list and notice the user, or, if the request has been successful, we print what IP icanhazip sees.

# Make the call
    try:
      my_ip = urlopen(req).read().decode('utf8')
      print('#' + str(n) + ': ' + my_ip)
    except: # If error, delete this proxy and find another one
      del proxies[proxy_index]
      print('Proxy ' + proxy['ip'] + ':' + proxy['port'] + ' deleted.')
      proxy_index = random_proxy()
      proxy = proxies[proxy_index]

Save the file (in my case scrape_me.py, just a tribute to "Rape me" of Nirvana) and execute it. You'll see something like this:

Code

This is the full code or, if you want, you can see it in this Github repository.

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
from fake_useragent import UserAgent
import random

ua = UserAgent() # From here we generate a random user agent
proxies = [] # Will contain proxies [ip, port]

# Main function
def main():
  # Retrieve latest proxies
  proxies_req = Request('https://www.sslproxies.org/')
  proxies_req.add_header('User-Agent', ua.random)
  proxies_doc = urlopen(proxies_req).read().decode('utf8')

  soup = BeautifulSoup(proxies_doc, 'html.parser')
  proxies_table = soup.find(id='proxylisttable')

  # Save proxies in the array
  for row in proxies_table.tbody.find_all('tr'):
    proxies.append({
      'ip':   row.find_all('td')[0].string,
      'port': row.find_all('td')[1].string
    })

  # Choose a random proxy
  proxy_index = random_proxy()
  proxy = proxies[proxy_index]

  for n in range(1, 100):
    req = Request('http://icanhazip.com')
    req.set_proxy(proxy['ip'] + ':' + proxy['port'], 'http')

    # Every 10 requests, generate a new proxy
    if n % 10 == 0:
      proxy_index = random_proxy()
      proxy = proxies[proxy_index]

    # Make the call
    try:
      my_ip = urlopen(req).read().decode('utf8')
      print('#' + str(n) + ': ' + my_ip)
    except: # If error, delete this proxy and find another one
      del proxies[proxy_index]
      print('Proxy ' + proxy['ip'] + ':' + proxy['port'] + ' deleted.')
      proxy_index = random_proxy()
      proxy = proxies[proxy_index]

# Retrieve a random index proxy (we need the index to delete it if not working)
def random_proxy():
  return random.randint(0, len(proxies) - 1)

if __name__ == '__main__':
  main()

Additional resources

Take a look at this awesome article by Zach Burchill about how to scrape a lot of data in Python.

It works great, but something's missing: the possibility to schedule a job to run it in a specific time, like the crontab and the "schedule" of Laravel.

You have two ways to achieve it:

1. Edit the crontab of your server (crontab -e) and add something like 15 * * * * cd /var/www/your_project && adonis yourcommand >/dev/null 2>&1 (just an example);

2. Use Adonis Scheduler, which is a third-party library (not official) but very useful.

The repository I linked is a fork of a fork. The original repo doesn't have the support for Adonis 4 (the latest version), so the user ntvsx193 updated it making it work with v4 and I created a "security" copy (forking its fork, yes).

Installation

Note: If you are using Adonis v3.x, please follow the original installation tutorial on the repository.

The first part of everything you do is the installation. Since its a node.js project, we use npm with this command: npm i --save git+https://github.com/danilopolani/adonis-scheduler.git. Note we used the syntax git+ because the updated version is not available on npm packages, so we link directly to the repository.

Now open your start/app.js file and...

1. Add adonis-scheduler/providers/SchedulerProvider to your providers array like this:

const providers = [
  ...
  'adonis-scheduler/providers/SchedulerProvider',
];

2. Create the alias:

const aliases = {
  ...
  Scheduler: 'Adonis/Addons/Scheduler',
};

3. Register the command:

const aceProviders = [
  ...
  'adonis-scheduler/providers/CommandsProvider',
];

Now let's write a task.

Tasks: schedule the world

The updates for our dear ntvsx193 include a command to generate a task; you need just to run adonis make:task ScheduleIt from your terminal and here you go. Note: The make:task command will strip the word "Task" from the name, so if you write "MyTask" it will create a file named only "My.js".

Open your generated file at app/Tasks/ScheduleIt.js and you'll see this structure:

'use strict'

const Task = use('Task');

class ScheduleIt extends Task {
  static get schedule() {
    return '0 */1 * * * *';
  }

  async handle() {
    this.info('Task ScheduleIt handle');
  }
}

module.exports = ScheduleIt;

In the schedule section, you have to put the crontab value. I suggest you to use the useful Crontab generator tool to know what to put in here.

For example, if I want to schedule my job at 3:15am of every day, so I choose the minute and the hour in the list at the right of each box, leaving Days, Months and Weekday set to "Every ...".

Write whatever you want under "Command to execute" and press Enter. It will print the complete line to copy-paste, for example 15 3 * * * foo >/dev/null 2>&1, where 'foo' is my fake command. We need to retrieve only the part with the asterisks: 15 3 * * *.

Putting it in the schedule function, it becomes this:

static get schedule() {
  return '15 3 * * *';
}

Since we need to test if it's working, we use it as "schedule time" every minute: * * * * *.

In the handle section, you'll put your real code. Since it's an async method, you can use as well the await functionality of ES6.

In this tutorial, we leave the this.info helper to print in our terminal, every minute, the phrase Task ScheduleIt handle.

Run the scheduler

The repository tells us only to run the command adonis run:scheduler, but of course, in our server we can't run it from the terminal, but we need something to keep it alive.

I have found three ways for this:

1. Use pm2, the "equivalent" of supervisor for node.js processes.

2. Use supervisor to keep the process alive.

3. Include the schedule runner in your project.

After thinking about them, I realized that the last one is the more simple to adopt: with pm2 you need to create a new process and save it (to preserve it when you reboot your server); with supervisor you need to create a config file for the process, reread and update.

With the last option you just need to copy and paste a code in a file. Simple, isn't it?

Open the file start/kernel.js and right after the line const Server = use ('Server'), put this:

/*
|--------------------------------------------------------------------------
| Run Scheduler
|--------------------------------------------------------------------------
|
| Run the scheduler on boot of the web sever.
|
*/
const Scheduler = use('Adonis/Addons/Scheduler');
Scheduler.run();

Now we are ready to test it.

Test

Run the app like always, with adonis serve --dev and wait one minute to see the first message.

After two minutes, your terminal will look like the screenshot below.

It's 2020 and having a cache it's today a required part of your stack. For example, you could cache your database response for X time (or until a new entry comes in) to stop doing a query (or more!) for every request.

Recently I had to implement it in Node.js and I've chosen to go with Redis. I was looking for a decent but simple-to-use library and I didn't find so much, so I decided to write my own: Redache.

Let's see how to implement it!

Installation

To install the library, just run from your terminal yarn add redache or, if you use npm: npm i redache --save.

Then, of course, you have to include it in your project. If you use ES6 features, you can use import:

import Redache from 'redache';

// ...

Otherwise, you can go with the CommonJS way:

const Redache = require('redache');

// ...

The last but not least step is to initialize it. This is a minimal configuration, but you can check out all the available parameters in the Redis official package: https://github.com/NodeRedis/node_redis#options-object-properties

const cache = new Redache({
    host: '127.0.0.1',
    port: 6379,
    password: 'foobar',
});

Save values

I wanted to keep the library as easy as possible. To store a value, you just have to call the .set() method providing a key, a value and a TTL (expiration time).
Usually with these libraries, you have to specify the TTL as a number representing the seconds, but guess what? I didn't want to make your life harder, but easier, so... You can choose between four methods to specify your expiration and the first one si my favorite.

1. Human readable TTL Just pass a format like number TIME_UNIT to set the expiration. It uses moment behind the scenes to "decode" it, so TIME_UNIT can be something like days, day, d, month and so on.

// It expires in 5 hours
await cache.set('key', 'value', '5 hours');

2. Date instance If you're more "old school" and you have a Date instance, don't worry, we accept it as well.

// It expires the 1st Jan 2050
await cache.set('key', 'value', new Date(2050, 1, 1));

3. moment instance We said it uses moment under the hood, so of course you can provide a moment object.

// It expires in 1 month
await cache.set('key', 'value', moment().add(1, 'month'));

4. Seconds And finally, if you're a real boss, you can pass a number representing the seconds. I don't know why you should, but ehy, up to you!

// It expires in 3600 seconds -> 1 hour
await cache.set('key', 'value', 3600);

Object(s) and Array(s)

Most of the time you want to save a whole object or array, but the native Redis functions won't let you do that. But we do. Just pass your desired variable and the library will take care of JSON serializing and deserializing it.

Retrieve values

After you saved your value you may desire (or may not?) to retrieve it.

const value = await cache.get('key');

As we said, Redache accepts objects and arrays too, let's see how:

await cache.set('key', { foo: 'bar' }, '1 hour');

// Retrieve it
// It will be: { foo: 'bar' } but stored as '{"foo":"bar"}'
const value = await cache.get('key');

Easy, right? Now let's say that you want to store a value if a key is not found (or expired) on Redis, you would write something like:

let value = await cache.get('key');
if (value === null) {
    value = 'my fallback';
    await cache.set('key', 'my fallback', '1 day');
}

NOOOOOOO!! The library can manage this for you, in just one line! In fact, the 2nd and 3rd arguments of .get() are, respectively, the fallback value and the fallback TTL.

const value = await cache.get('key', 'my fallback', '1 day');

Isn't it more simple? The fallback value accepts as well a function (async too):

const value = await cache.get('key', () => 'my fallback', '1 day');

// Async example
const value = await cache.get('key', () => Promise.resolve('my fallback'), '1day ');

Delete a key

Like alchemy, you can create and destroy (yes, I recently saw Fullmetal Alchemist). Just invoke .forget() method specifying your key.

// Save a record
await cache.set('key', 'value', '1 day');

// It returns `value`
const value = await cache.get('key');

// Now let's remove it
await cache.forget('key');

// It will return null now, because we removed it
const valueAgain = cache.get('key');

Conclusion

That's it! I hope this library can help you as much as it helped me and don't forget to leave a star!

Star

You are creating your hybrid app and when your user deletes an item from the cart you want to show a toast with the confirmation message and a button to undo the action, but in the documentation there's no way to customize the callback, so how to do this?

My boring problem and the way to solve it

This was, more or less, my case. After a lot of searches, and a nice headache - maybe it was only the air conditioner - when I decided to renounce finally the answer!

Things to know about how to solve it:

1. The ToastController class has a dismiss method;

2. The dismiss method callback has two arguments, and not one.

Why this is so important? Because without the second parameter, which specifies the motivation of the dismiss: the callback is invoked when the toast disappears automatically after the specified time or when the user clicks the button, there was no way to understand when was the button and when the timeout was.

But stop talking and start coding.

Code!

First of all, include ToastController in your .ts page:

import { ToastController } from 'ionic-angular';

And include it in your constructor:

constructor(public toastCtrl: ToastController) {

Now we can create the toast, like the documentation, customize also the close button message.

// Create a toast
let toast = this.toastCtrl.create({
    message: 'Item removed from the cart',
    duration: 3000,
    showCloseButton: true,
    closeButtonText: 'Undo',
});

I was talking about the dismiss callback, so let's write it.

// Handle "undo" action
toast.onDidDismiss((data, role) => {
    //
});

As you can see, there's a second parameter called role and with it, we can execute our code only when the user hits the button.

// Handle "undo" action
toast.onDidDismiss((data, role) => {
    if (role == 'close') {
        yourRecoveringMethod();
    }
});

And, finally, let's show the toast with toast.present();. The final code will look like this:

// Create a toast
let toast = this.toastCtrl.create({
    message: 'Item removed from the cart',
    duration: 3000,
    showCloseButton: true,
    closeButtonText: 'Undo',
});

// Handle "undo" action
toast.onDidDismiss((data, role) => {
    if (role == 'close') {
        yourRecoveringMethod();
    }
});

toast.present();

This was so simple, why not include it in the documentation? This will be a mystery.