Updateing elasticsearch to 2.0 and de-dotting es indexes [FIX]

Updating elasticsearch to a newer version has been quite a breeze in the past. But with the arrival of the new major es-release 2.0, I thought a bit more thorough test were in order. A good read for breaking changes can be found here. A very useful es-plugin to check your existing indexes for compatibility issues can be found here.

One of the major changes that did hit us, is the fact that field names may no longer contain dots. We use the elk stack to mainly log webserver logs. For some log types we also parse the URL get params to param_name => param_value pairs. Sadly, some of those param_names contain a dot, e.g. „document.x=123&document.y=345“. So, to migrate the exiting indexes we need to get rid of the dots in those fields.

Here is the LumberMill configuration that will reindex and replace dots with underscores:

# Sets number of parallel LumberMill processes.
- Global:
   workers: 3

- ElasticSearch:
   nodes: ['ELASTICSEARCH_HOST:9200']
   search_type: scan
   index_name: 'INDEX_NAME'

# Recursively replace dots with underscores in all fieldnames below field "params".
- ModifyFields:
   action: rename_replace
   source_field: params
   recursive: True
   old: .
   new: _

# Copy old event type to new event.
- ModifyFields:
   action: insert
   target_field: lumbermill.event_type
   value: $(_type)

# Drop internal es fields prior to reindex.
- ModifyFields:
   action: delete
   source_fields: ['_uid', '_id', '_type', '_source', '_all', '_parent', '_field_names', '_routing', '_index', '_size', '_timestamp', '_ttl', '_score']

- SimpleStats

- ElasticSearchSink:
   nodes: ['ELASTICSEARCH_HOST:9200']
   index_name: 'NEW_INDEX_NAME'
Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar

git-http-backend with CentOS7.1 and Apache 2.4 [FIX]

To enable anonymous read access but authenticated write access, require authorization for both the initial ref advertisement (which we detect as a push via the service parameter in the query string), and the receive-pack invocation itself:

<VirtualHost *:80>
        ServerName <your server name>
	DocumentRoot </path/to/git/repositories>
	SetEnv GIT_PROJECT_ROOT /path/to/git/repositories
	SetEnv GIT_HTTP_EXPORT_ALL
	SetEnv REMOTE_USER $REDIRECT_REMOTE_USER
	
	AliasMatch ^/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$          /opt/repositories/git/$1
	AliasMatch ^/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /opt/repositories/git/$1
	
	ScriptAlias / /usr/libexec/git-core/git-http-backend/
        <Directory /usr/libexec/git-core*> 
        	Options +Indexes +ExecCGI 
           	AllowOverride None 
    		Require all granted
        </Directory> 

	<Directory "/path/to/git/repositories">
		Dav On
	</Directory>

	#Authenticated commits.
	<LocationMatch "^/">
        	AuthType Basic
                AuthName "git access"
                AuthBasicProvider ldap
                AuthUserFile /dev/null
                AuthLDAPUrl "ldap URI" TLS
		# As of commit 986bbc0 (http: don't always prompt for password, 2011-11-04)
		# git sends a simple get request prior to sending the whole push packet to check if it needs 
		# authorization. See also: http://git.661346.n2.nabble.com/git-no-longer-prompting-for-password-td7565755.html
		<if "%{QUERY_STRING} =~ /service=git-receive-pack/">
                	Require valid-user
		</if>	
		<elseif "%{REQUEST_URI} =~ /^git-receive-pack$/">
                        Require valid-user
		</elseif>
	</LocationMatch>
</VirtualHost>
Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar

nginx 1.8 not caching reverse proxy [FIX]

Just updated our loadbalancers to nginx-1.8. As we need some custom modules, I grabbed the SRPM from here and modified the spec file to fit our needs. As usual, compiling was a breeze and I updated the staging server. Everything was fine, except that nginx was not caching some of our backends responses. A quick check with the live nginx servers showed that this problem was introduced by the update. So I checked which calls did not get cached and had a closer look at the response headers from the backends and it turned out, that the vary-http header was the only difference. Headers like:

< HTTP/1.1 200 OK
< Content-Length: 22333
< Content-Language: *
< Content-Type: application/xml; charset=UTF-8
< Expires: Wed, 09 Sep 2015 13:15:38 GMT
< ETag: "8eb73cc447dc29e1c3620c747ba89b95"
< Cache-Control: max-age=600, max-age=600
< Date: Wed, 09 Sep 2015 13:05:38 GMT
< Accept-Ranges: bytes
< Vary: Accept-Charset, Accept-Encoding, Accept-Language, Accept

would cause the response to not be cached but headers like:

< HTTP/1.1 200 OK
< Content-Length: 22333
< Content-Language: *
< Content-Type: application/xml; charset=UTF-8
< Expires: Wed, 09 Sep 2015 13:15:38 GMT
< ETag: "8eb73cc447dc29e1c3620c747ba89b95"
< Cache-Control: max-age=600, max-age=600
< Date: Wed, 09 Sep 2015 13:05:38 GMT
< Accept-Ranges: bytes
< Vary: Accept

would not cause this problem. So I suspected that the length of the vary header might be the problem. Wrote a simple tcp server to check and – well what should I say: the answer was 42. That is the maximum length of the vary header in this nginx release.
A quick look at the source provided more insight. In src/http/ngx_http_cache.h, line 28:

#define NGX_HTTP_CACHE_VARY_LEN      42

So i patched this to 250, recompiled and the problem was fixed.

Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar

Noch is Sommer…







Veröffentlicht unter /dev/video | Hinterlasse einen Kommentar

HA nfs server with drbd, zfs and heartbeat on CentOS 6.6 [FIX]

Following this how-to to set up a ha nfs server I stumbled into a problem with zfs and the takeover process of heartbeat.
When forcing the active machine to standby via:

/usr/share/heartbeat/hb_standby

the takeover failed with:

...
ResourceManager(default)[3439]:	2015/07/13_16:22:25 info: Releasing resource group: distfs1-test IPaddr::172.16.10.10/24/eth0 drbddisk::data zfs nfs
ResourceManager(default)[3439]:	2015/07/13_16:22:25 info: Running /etc/init.d/nfs  stop
ResourceManager(default)[3439]:	2015/07/13_16:22:26 info: Running /etc/init.d/zfs  stop
ResourceManager(default)[3439]:	2015/07/13_16:22:26 info: Running /etc/ha.d/resource.d/drbddisk data stop
ResourceManager(default)[3439]:	2015/07/13_16:22:28 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk
ResourceManager(default)[3439]:	2015/07/13_16:22:29 info: Retrying failed stop operation [drbddisk::data]
ResourceManager(default)[3439]:	2015/07/13_16:22:29 info: Running /etc/ha.d/resource.d/drbddisk data stop
ResourceManager(default)[3439]:	2015/07/13_16:22:31 ERROR: Return code 1 from /etc/ha.d/resource.d/drbddisk
...

Further tests produced a bit more detailed error:

[root@distfs1-test ~]# /etc/ha.d/resource.d/drbddisk data stop
1: State change failed: (-12) Device is held open by someone
Command 'drbdsetup-84 secondary 1' terminated with exit code 11
1: State change failed: (-12) Device is held open by someone
Command 'drbdsetup-84 secondary 1' terminated with exit code 11

Investigating further showed that even after an:

/etc/init.d/zfs stop

There was still a zfs process running:

[root@distfs1-test ~]# ps xa| grep zfs
 4021 ?        S<     0:00 [zfs_iput_taskq/]

After removing the kernel module by hand:

[root@distfs1-test ~]# rmmod zfs
[root@distfs1-test ~]# /etc/ha.d/resource.d/drbddisk data stop
[root@distfs1-test ~]# echo $?
0

everything worked fine.

I ended up fixing the zfs Sys-V init script simply by applying this patch:

112a113,115
> 	# Unload kernel module. A zfs_iput_taskq will still be running, even
> 	# after calling /etc/init.d/zfs stop. Removing the kernel module resolves this issue.
> 	/sbin/rmmod zfs
Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar

Simple benchmark of LumberMill

Being curious as to how well LumberMill scales with more CPUs, I did a quick and simple benchmark. And while I was at it, I also included performance data for running LumberMill with Python-2.6, pypy-2.5.1 and the new pypy-4.0. As with all benchmarks, results needs to be taken with a pillar of salt ;)

The machine used was a x86_64 CentOS-6.5 vm image with 1GB RAM and 3 cores. The host was a Core i3 iMac/3.06GHz.

The logstash data is only included as a reference for me. The generator.threads setting was set to the number of filter workers started for each run.

Here are the results:

Logstash-2.1.0, LumberMill Python-2.6, LumberMill pypy-2.6, LumberMill pypy-4.0

The LumberMill configuration used:

# Sets number of parallel LumberMill processes.
- Global:
   workers: 3

- Spam:
   event: '<13>Feb 11 13:01:36 john nginx: test 1392123696.090 localhost GET "/item/get?Template=Mobile_FacetedBrowsing_Bereich_DE&RowCount=24&Query=Binaries_vorderansicht:%3A%5B0+TO+A%5D&Sort=Preis_ASC&StartAtRowNo=265&Bereich=%22Skate%22&Oberkategorie=%22Longboards%22&Kundeninfo=%22Lieferbar%22coming+soon%22" "" - 200 98198 403 0.002 127.0.0.1 57679 "-" "/index.php" "curl/7.15.5 libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 server/localhost" HIT "-" "-" "application/xml" "deflate, gzip" "gzip" "client-user-agent\x02Mozilla/5.0 (Linux; U; Android 4.1.2; de-de; GT-I8190 Build/JZO54K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30\x01client-addr\x02127.0.0.1,13.13.13.25\x01server-name\x02localhost"'
   #events_count: 105

- RegexParser:
   source_field: data
   hot_rules_first: True
   field_extraction_patterns:
    - test_access_log: '(?P<syslog_prival>\<\d+\>)(?P<log_timestamp>\w+\s+\d+\s+\d+:\d+:\d+)\s+(?P<host>[\w\._\-]+)\s+nginx: test (?P<request_ts>\d+\.\d+) (?P<server_name>[\w\._\-]+) (?P<http_method>[\w\-]+) \"(?P<uri>[^\"]*)\" \"(?P<request_body>[^\"]*)\" (?P<username>[\w\-]+) (?P<http_status>\d+) (?P<bytes_sent>\d+) (?P<request_size>\d+) (?P<request_time>[\d\.]+) (?P<remote_ip>[\d\.]+) (?P<remote_port>\d+) \"(?P<x_forwarded_for>[^\"]*)\" \"(?P<referer>[^\"]*)\" \"(?P<user_agent>[^\"]*)\" (?P<cache_status>[\w\-]+) \"(?P<upstream_response_time>[^\"]*)\" \"(?P<upstream_addr>[^\"]*)\" \"(?P<content_type>[^\"]*)\" \"(?P<accept_encoding>[^\"]*)\" \"(?P<content_encoding>[^\"]*)\" \"(?P<custom_vars>[^\"]*)\"'

# Print out some stats every 10 seconds.
- SimpleStats:
   interval: 10

- AddDateTime:
   format: '%Y-%m-%dT%H:%M:%S.%f'
   target_field: "@timestamp"

# Add geo info based on the lookup_fields. The first field in <source_fields> that yields a result from geoip will be used.
- AddGeoInfo:
   geoip_dat_path: /usr/share/GeoIP/GeoLiteCity.dat
   source_fields: [x_forwarded_for, remote_ip]
   geo_info_fields: ['latitude', 'longitude', 'country_code']

# Nginx logs request time in seconds with milliseconds as float. Apache logs microseconds as int.
# At least cast nginx to integer.
- Math:
   filter: if $(server_type) == "nginx"
   target_field: request_time
   function: float($(request_time)) * 1000

- ModifyFields:
   action: cast_to_int
   source_fields: [http_status, bytes_sent, request_size, request_time, remote_port, request_ts,]

- ModifyFields:
   action: cast_to_float
   source_fields: [upstream_response_time, latitude, longitude]

# Replace custom_vars Separators.
- ModifyFields:
   action: string_replace
   source_field: custom_vars
   old: '\x01'
   new: '¶'

# Replace custom_vars Separators.
- ModifyFields:
   action: string_replace
   source_field: custom_vars
   old: '\x02'
   new: '='

# Split custom vars to map.
- ModifyFields:
   action: key_value
   line_separator: '¶'
   kv_separator: '='
   source_field: custom_vars

- ModifyFields:
   filter: if $(http_status)
   action: map
   source_field: http_status
   map: {100: 'Continue', 200: 'OK', 301: 'Moved Permanently', 302: 'Found', 304: 'Not Modified', 400: 'Bad Request', 401: 'Unauthorized', 403: 'Forbidden', 404: 'Not Found', 500: 'Internal Server Error', 502: 'Bad Gateway'}

# Kibana’s ‘bettermap’ panel needs an array of floats in order to plot events on map.
- ModifyFields:
   filter: if $(latitude)
   action: merge
   source_fields: [longitude, latitude]
   target_field: geoip

- UserAgentParser:
   source_fields: user_agent

- DevNullSink

The logstash configuration:

input {
	generator {
	    threads => 3
		lines => ['<13>Feb 11 13:01:36 john nginx: test 1392123696.090 localhost GET "/item/get?Template=Mobile_FacetedBrowsing_Bereich_DE&RowCount=24&Query=Binaries_vorderansicht:%3A%5B0+TO+A%5D&Sort=Preis_ASC&StartAtRowNo=265&Bereich=%22Skate%22&Oberkategorie=%22Longboards%22&Kundeninfo=%22Lieferbar%22coming+soon%22" "" - 200 98198 403 0.002 127.0.0.1 57679 "-" "/index.php" "curl/7.15.5 libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 server/localhost" HIT "-" "-" "application/xml" "deflate, gzip" "gzip" "client-user-agent\x02Mozilla/5.0 (Linux; U; Android 4.1.2; de-de; GT-I8190 Build/JZO54K) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30\x01client-addr\x02127.0.0.1,13.13.13.25\x01server-name\x02localhost"']
	}
}

filter {
        grok {
            match => [ "message", '(?<syslog_prival>\<\d+\>)(?<log_timestamp>\w+\s+\d+\s+\d+:\d+:\d+)\s+(?<host>[\w\._\-]+)\s+nginx: test (?<request_ts>\d+\.\d+) (?<server_name>[\w\._\-]+) (?<http_method>[\w\-]+) \"(?<uri>[^\"]*)\" \"(?<request_body>[^\"]*)\" (?<username>[\w\-]+) (?<http_status>\d+) (?<bytes_sent>\d+) (?<request_size>\d+) (?<request_time>[\d\.]+) (?<remote_ip>[\d\.]+) (?<remote_port>\d+) \"(?<x_forwarded_for>[^\"]*)\" \"(?<referer>[^\"]*)\" \"(?<user_agent>[^\"]*)\" (?<cache_status>[\w\-]+) \"(?<upstream_response_time>[^\"]*)\" \"(?<upstream_addr>[^\"]*)\" \"(?<content_type>[^\"]*)\" \"(?<accept_encoding>[^\"]*)\" \"(?<content_encoding>[^\"]*)\" \"(?<custom_vars>[^\"]*)\"']
        }

        # Add geoinfo to custom vars.
        geoip	{
            source => "[remote_ip]"
            target => "geoinfo"
            database => "/opt/logstash-1.4.2/vendor/geoip/GeoLiteCity.dat"
        }

        # Nginx logs request time in seconds with milliseconds as float. Apache logs microseconds as int.
        ruby { code => "begin event['request_time'] = (event['request_time'] * 1000).ceil; rescue; end" }

        mutate {
            # Cast fields.
            # Int: [http_status, bytes_sent, remote_port, request_time, request_size, TreeNodeID]
            convert =>  [ "http_status", "integer" ]
            convert =>  [ "bytes_sent", "integer" ]
            convert =>  [ "request_size", "integer" ]
            convert =>  [ "request_time", "float" ]
            convert =>  [ "remote_port", "integer" ]
            convert =>  [ "request_ts", "integer" ]
            # Float: [upstream_response_time, latitude, longitude]
            convert =>  [ "upstream_response_time", "float" ]
            convert =>  [ "latitude", "float" ]
            convert =>  [ "longitude", "float" ]
            # CustomVars Separators
            gsub => ["custom_vars", "\\x01", '¶']
            gsub => ["custom_vars", "\\x02", '=']
        }

        # Split custom_vars to map.
        kv {
            source => "custom_vars"
            field_split => "¶"
            target => custom_vars
        }

        translate {
            field => "http_status"
            destination => "http_status_mapped"
            override => true
            dictionary => [ '100', 'Continue',
                            '200', 'OK',
                            '301', 'Moved Permanently',
                            '302', 'Found',
                            '304', 'Not Modified',
                            '400', 'Bad Request',
                            '401', 'Unauthorized',
                            '403', 'Forbidden',
                            '404', 'Not Found',
                            '500', 'Internal Server Error',
                            '502', 'Bad Gateway']
        }

        if [custom_vars][client-user-agent] {
            useragent {
                source => "[custom_vars][client-user-agent]"
                target => "custom_vars"
                prefix => "useragent_"
            }
        }

        metrics {
    		meter => "events"
    	    add_tag => "metric"
  	    }
}

output {
	if 'metric' in [tags]   {
        	stdout {
		        codec => rubydebug
	    	}
	}
}
Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar

Americana







Veröffentlicht unter /dev/video | Hinterlasse einen Kommentar

Folk








Veröffentlicht unter /dev/video | Hinterlasse einen Kommentar

Saturday night







Veröffentlicht unter /dev/video | Hinterlasse einen Kommentar

Python pcapy, pypy and „undefined symbol: __gxx_personality_v0“ [FIX]

Got this error while trying to use this package with pypy-2.4
Solved it by downloading the tar from here and building/installing it with:

PYTHONPATH=/usr/lib64/pypy-2.4/site-packages;CFLAGS="-lstdc++" /usr/lib64/pypy-2.4/bin/pypy setup.py install
Veröffentlicht unter /dev/administration | Hinterlasse einen Kommentar