Installing Python into your home directory

Sometimes we may need a newer Python version than the one installed into our station, and the Linux distribution does not provide the last release in its repositories. We could get rid the installed Python packages and make a new installation from the source code. However, various OS applications make use of the default Python version installed, thus there is a risk of breaking some applications or services of the system in the case of overriding the Python provided by the package management system used, yum or apt, for example.

Thus, we may install Python into the local user directory, and let the default Python installation intact.

Another reason to do this is to have multiple Python environments when developing or debugging old Python scripts.

The following commands were tested into an Ubuntu 16.04 amd64 docker container.

Install prerequisites for downloading and compiling Python from source:

# apt install xz-utils wget gcc make \
  libssl-dev libffi-dev zlib1g-dev

If you are running CentOS 7,  install the following packages:

# yum groupinstall -y "development tools"

# yum install -y \
  libffi-devel \
  zlib-devel \
  bzip2-devel \
  openssl-devel \
  ncurses-devel \
  sqlite-devel \
  readline-devel \
  tk-devel \
  gdbm-devel \
  db4-devel \
  libpcap-devel \
  xz-devel \
  expat-devel

Create a folder for the new Python installation, download, and extract the source code. In this example, I used the hidden directory $HOME/.python.

mkdir $HOME/.python
cd $HOME/.python
wget https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz
tar Jxvf Python-3.7.0.tar.xz
cd Python-3.7.0

Compile and install Python.

./configure --prefix=$HOME/.python --enable-optimizations
make
make install

Next, we need to edit $HOME/.profile , and $HOME/.bashrc files to add the Python’s bin directory to $PATH.

export PATH="$HOME/.python/bin:$PATH"
export PYTHONPATH="$HOME/.python"

Then we can reload the current session by doing:

source $HOME/.profile

Now we have multiple python installations on our system. If we type ‘python’ and press TAB twice we can see:

roger@9e65d2dd5598:~$ python
python python2.7 python3-config python3.5m python3.7-config python3.7m-config
python2 python3 python3.5 python3.7 python3.7m python3m

So, to run the new python REPL, we must enter:

roger@9e65d2dd5598:~$ python3.7
Python 3.7.0 (default, Sep 26 2018, 22:06:04)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Pip also is referred by its version:

roger@9e65d2dd5598:~$ pip3.7

Usage:
pip3.7 <command> [options]

When developing executable python scripts for this environment, the bash shebang used must be like:

#!/usr/bin/env python3.7

Delete the Python source code and compilation directory:

rm -fr $HOME/.python/Python-3.7.0*

 

This article is adapted from the solution presented in http://thelazylog.com/install-python-as-local-user-on-linux/.

“ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network” – docker-compose and bridge config problem.

Recently I had an error while running docker-compose.

$ docker-compose up -d --build
Creating network "courseenvironments_default" with the default driver
ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

After googled a bit, I tried to remove unused docker networks with:

$ docker network prune

But, no success.

After inspecting my /etc/network/interfaces file, I decided to change my network configuration:

# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto br0
iface br0 inet dhcp
             bridge_ports eth0
             bridge_stp off
             bridge_fd 0
             post-up vconfig add br0 100
             post-up ifconfig br0.100 200.200.200.200/23
             post-up route add default gw 200.200.200.1
             post-up route del default gw 10.1.1.1
             post-up route add -net 10.0.0.0/8 gw 10.1.1.1
             post-up route add -net 192.168.0.0/16 gw 10.1.1.1
             post-up route add -net 172.16.0.0/12 gw 10.1.1.1
             post-down vconfig rem br0.100

I used to use this bridge in my old VirtualBox and GNS3 configurations; thus I removed it and used my VLAN tagged config directly into eth0 interface:

auto eth0
iface eth0 inet dhcp
            post-up vconfig add eth0 100
            post-up ifconfig eth0.100 200.200.200.200/23
            post-up route add default gw 200.200.200.1
            post-up route del default gw 10.1.1.1
            post-up route add -net 10.0.0.0/8 gw 10.1.1.1
            post-up route add -net 192.168.0.0/16 gw 10.1.1.1
            post-up route add -net 172.16.0.0/12 gw 10.1.1.1
            post-down vconfig rem eth0.100

After stopping the networking and applying the new conf, docker-compose could create the new network, and the build process finished without errors:

# ifdown br0
# vconfig rem br0.100
# brctl delbr br0
# service networking restart

$ docker-compose up -d --build
Building jenkinsBuilding jenkins
Step 1/8 : FROM jenkins:latest
 ---> cd14cecfdb3a
Step 2/8 : MAINTAINER Ernest Mueller <ernestmueller@theagileadmin.com>
 ---> Using cache
 ---> bc8067b1e3b2
Step 3/8 : USER root
 ---> Using cache
 ---> b0de05eb295c
Step 4/8 : RUN     apt-get update &&     apt-get install -y build-essential &&     apt-get clean &&     rm -rf /var/lib/apt/lists/*

Step 1/8 : FROM jenkins:latest
 ---> cd14cecfdb3a
Step 2/8 : MAINTAINER Ernest Mueller <ernestmueller@theagileadmin.com>
 ---> Using cache
 ---> bc8067b1e3b2
Step 3/8 : USER root
 ---> Using cache
 ---> b0de05eb295c
Step 4/8 : RUN     apt-get update &&     apt-get install -y build-essential &&     apt-get clean &&     rm -rf /var/lib/apt/lists/*

Realizando particionamento de tabelas para melhora de performance das consultas do Zabbix 2.0.5 e PostgreSQL 9.0.1

Recentemente atualizei meu NMS de Zabbix 1.8.2 (Debian Squeeze) para o 2.0.5, um dos principais motivos que me levou a esta versão foi poder realizar o particionamento de tabelas, já que o desempenho no meu cenário (128 hosts monitorados, retenção de um ano e NVPS=110) estava começando a ficar sofrível em alguns momentos.

O propósito do particionamento de tabelas é dividir a tabela history do zabbix em intervalos pré-definidos, como diário, semanal ou mensal. A escolha do intervalo a ser utilizado deve se basear na quantidade de dados inserido no banco em determinado período.

Após o particionamento será criado um novo schema onde ficaram as tabelas contendo os dados separados pelo intervalo selecionado, isso facilita bastante a busca das informações na base.

Todos os passos são realizados no PostgreSQL, tornando a solução totalmente transparente ao Zabbix.

Criação do novo schema.


CREATE SCHEMA partitions
AUTHORIZATION zabbix;

Criar a função que cria as partições:


-- Function: trg_partition()

-- DROP FUNCTION trg_partition();

CREATE OR REPLACE FUNCTION trg_partition()
RETURNS TRIGGER AS
$BODY$
DECLARE
prefix text := 'partitions.';
timeformat text;
selector text;
_interval INTERVAL;
tablename text;
startdate text;
enddate text;
create_table_part text;
create_index_part text;
BEGIN

selector = TG_ARGV[0];

IF selector = 'day' THEN
timeformat := 'YYYY_MM_DD';
ELSIF selector = 'month' THEN
timeformat := 'YYYY_MM';
END IF;

_interval := '1 ' || selector;
tablename :=  TG_TABLE_NAME || '_p' || TO_CHAR(TO_TIMESTAMP(NEW.clock), timeformat);

EXECUTE 'INSERT INTO ' || prefix || quote_ident(tablename) || ' SELECT ($1).*' USING NEW;
RETURN NULL;

EXCEPTION
WHEN undefined_table THEN

startdate := EXTRACT(epoch FROM date_trunc(selector, TO_TIMESTAMP(NEW.clock)));
enddate := EXTRACT(epoch FROM date_trunc(selector, TO_TIMESTAMP(NEW.clock) + _interval ));

create_table_part:= 'CREATE TABLE IF NOT EXISTS '|| prefix || quote_ident(tablename) || ' (CHECK ((clock >= ' || quote_literal(startdate) || ' AND clock < ' || quote_literal(enddate) || '))) INHERITS ('|| TG_TABLE_NAME || ')';
create_index_part:= 'CREATE INDEX '|| quote_ident(tablename) || '_1 on ' || prefix || quote_ident(tablename) || '(itemid,clock)';

EXECUTE create_table_part;
EXECUTE create_index_part;

--insert it again
EXECUTE 'INSERT INTO ' || prefix || quote_ident(tablename) || ' SELECT ($1).*' USING NEW;
RETURN NULL;

END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION trg_partition()
OWNER TO postgres;

Criar as triggers para cada tabela que se queira particionar e o intervalo de particionamento – diário ou mensal.


CREATE TRIGGER partition_trg BEFORE INSERT ON history           FOR EACH ROW EXECUTE PROCEDURE trg_partition('day');
CREATE TRIGGER partition_trg BEFORE INSERT ON history_sync      FOR EACH ROW EXECUTE PROCEDURE trg_partition('day');
CREATE TRIGGER partition_trg BEFORE INSERT ON history_uint      FOR EACH ROW EXECUTE PROCEDURE trg_partition('day');
CREATE TRIGGER partition_trg BEFORE INSERT ON history_str_sync  FOR EACH ROW EXECUTE PROCEDURE trg_partition('day');
CREATE TRIGGER partition_trg BEFORE INSERT ON history_log       FOR EACH ROW EXECUTE PROCEDURE trg_partition('day');
CREATE TRIGGER partition_trg BEFORE INSERT ON trends            FOR EACH ROW EXECUTE PROCEDURE trg_partition('month');
CREATE TRIGGER partition_trg BEFORE INSERT ON trends_uint       FOR EACH ROW EXECUTE PROCEDURE trg_partition('month');

Manutenção da retenção e remoção de partições antigas

A seguinte função é utilizada para remover partições antigas e poderá ser agendada para rodar via cron.

-- Function: delete_partitions(interval, text)-- DROP FUNCTION delete_partitions(interval, text);

CREATE OR REPLACE FUNCTION delete_partitions(intervaltodelete INTERVAL, tabletype text)
RETURNS text AS
$BODY$
DECLARE
result RECORD ;
prefix text := 'partitions.';
table_timestamp TIMESTAMP;
delete_before_date DATE;
tablename text;

BEGIN
FOR result IN SELECT * FROM pg_tables WHERE schemaname = 'partitions' LOOP

table_timestamp := TO_TIMESTAMP(substring(result.tablename FROM '[0-9_]*$'), 'YYYY_MM_DD');
delete_before_date := date_trunc('day', NOW() - intervalToDelete);
tablename := result.tablename;

-- Was it called properly?
IF tabletype != 'month' AND tabletype != 'day' THEN
RAISE EXCEPTION 'Please specify "month" or "day" instead of %', tabletype;
END IF;

--Check whether the table name has a day (YYYY_MM_DD) or month (YYYY_MM) format
IF LENGTH(substring(result.tablename FROM '[0-9_]*$')) = 10 AND tabletype = 'month' THEN
--This is a daily partition YYYY_MM_DD
-- RAISE NOTICE 'Skipping table % when trying to delete "%" partitions (%)', result.tablename, tabletype, length(substring(result.tablename from '[0-9_]*$'));
CONTINUE;
ELSIF LENGTH(substring(result.tablename FROM '[0-9_]*$')) = 7 AND tabletype = 'day' THEN
--this is a monthly partition
--RAISE NOTICE 'Skipping table % when trying to delete "%" partitions (%)', result.tablename, tabletype, length(substring(result.tablename from '[0-9_]*$'));
CONTINUE;
ELSE
--This is the correct table type. Go ahead and check if it needs to be deleted
--RAISE NOTICE 'Checking table %', result.tablename;
END IF;

IF table_timestamp <= delete_before_date THEN
RAISE NOTICE 'Deleting table %', quote_ident(tablename);
EXECUTE 'DROP TABLE ' || prefix || quote_ident(tablename) || ';';
END IF;
END LOOP;
RETURN 'OK';

END;

$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION delete_partitions(INTERVAL, text)
OWNER TO postgres;

Exemplos de chamada da função:


SELECT delete_partitions('7 days', 'day')
SELECT delete_partitions('11 months', 'month')