• README-JA(日本語)
• What is Sisimai
  • Key features
  • Command line demo
• Setting Up Sisimai
  • System requirements
  • Install
    • From RubyGems
    • From GitHub
• Usage
  • Basic usage
  • Convert to JSON
  • Read bounce object
  • Callback feature
  • One-Liner
  • Output example
• Sisimai Specification
  • Differences between Ruby version and Perl version
  • Other specification of Sisimai
• Contributing
  • Bug report
  • Emails could not be parsed
• Other Information
  • Related sites
  • See also
• Author
• Copyright
• License

What is Sisimai

Sisimai is a Ruby library for analyzing RFC5322 bounce emails and generating structured data from parsed results. The Ruby version of Sisimai is ported from the Perl version of Sisimai at github.com/sisimai/p5-Sisimai.

Key Features

• Convert Bounce Mails to Structured Data
  • Supported formats are Ruby(Hash, Array) and JSON(String)
• Easy to Install, Use.
  • gem install
  • git clone & make
• High Precision of Analysis
  • 2 times higher than bounceHammer
  • Support 23 known MTAs and 5 unknown MTAs
  • Support 22 major MSPs(Mail Service Providers)
  • Support 2 major Cloud Email Delivery Services(JSON format)
  • Support Feedback Loop Message(ARF)
  • Can detect 29 error reasons

Command line demo

The following screen shows a demonstration of Sisimai at the command line using Ruby(rb-Sisimai) and Perl(p5-Sisimai) version of Sisimai.

Setting Up Sisimai

System requirements

More details about system requirements are available at Sisimai | Getting Started page.

• Ruby 2.1.0 or later
  • Oj | The fastest JSON parser and object serializer
• Also works on JRuby 9.0.4.0 or later
  • JrJackson | A mostly native JRuby wrapper for the java jackson json processor jar

Install

From RubyGems

$ sudo gem install sisimai
Fetching: sisimai-4.23.0.gem (100%)
Successfully installed sisimai-4.23.0
Parsing documentation for sisimai-4.23.0
Installing ri documentation for sisimai-4.23.0
Done installing documentation for sisimai after 6 seconds
1 gem installed

From GitHub

$ cd /usr/local/src
$ git clone https://github.com/sisimai/rb-Sisimai.git
$ cd ./rb-Sisimai
$ sudo make depend install-from-local
gem install bundle rake rspec coveralls
...
4 gems installed
bundle exec rake install
sisimai 4.23.0 built to pkg/sisimai-4.23.0.gem.
sisimai (4.23.0) installed.

Usage

Basic usage

make() method provides feature for getting parsed data from bounced email messages like following.

#! /usr/bin/env ruby
require 'sisimai'
v = Sisimai.make('/path/to/mbox')       # or path to Maildir/

# Beginning with v4.23.0, both make() and dump() method of Sisimai class can
# read bounce messages from variable instead of a path to mailbox
f = File.open('/path/to/mbox', 'r');    # or path to Maildir/
v = Sisimai.make(f.read)

# If you want to get bounce records which reason is "delivered", set "delivered"
# option to make() method like the following:
v = Sisimai.make('/path/to/mbox', delivered: true)

if v.is_a? Array
  v.each do |e|
    puts e.class                # Sisimai::Data
    puts e.recipient.class      # Sisimai::Address
    puts e.timestamp.class      # Sisimai::Time

    puts e.addresser.address    # [email protected] # From
    puts e.recipient.address    # [email protected]   # To
    puts e.recipient.host       # example.jp
    puts e.deliverystatus       # 5.1.1
    puts e.replycode            # 550
    puts e.reason               # userunknown

    h = e.damn                  # Convert to HASH
    j = e.dump('json')          # Convert to JSON string
    puts e.dump('json')         # JSON formatted bounce data
  end
end

Convert to JSON

Sisimai.dump() method provides feature for getting parsed data as JSON string from bounced email messages like following.

# Get JSON string from parsed mailbox or Maildir/
puts Sisimai.dump('/path/to/mbox')  # or path to Maildir/

# dump() method also accepts "delivered" option like the following code:
puts Sisimai.dump('/path/to/mbox', delivered: true)

Read bounce object

The way to read a bounce object retrived from Cloud Email Services as JSON using their API is the following code. This feature is available at Sisimai 4.20.0 or later.

#! /usr/bin/env ruby
require 'json'
require 'sisimai'

j = JSON.load('{"notificationType"=>"Bounce", "bounce"=>{"...') # JSON String
v = Sisimai.make(j, input: 'json')

if v.is_a? Array
  v.each do |e|
    ...
  end
end

As of present, Only Amazon SES and SendGrid are supported.

Callback feature

Beginning with Sisimai 4.19.0, make() and dump() methods of Sisimai accept a Lambda (Proc object) in hook argument for setting a callback method and getting the results generated by the method via Sisimai::Data.catch method.

#! /usr/bin/env ruby
require 'sisimai'
callbackto = lambda do |v|
  r = { 'x-mailer' => '', 'queue-id' => '' }

  if cv = v['message'].match(/^X-Postfix-Queue-ID:\s*(.+)$/)
    r['queue-id'] = cv[1]
  end
  r['x-mailer'] = v['headers']['x-mailer'] || ''
  return r
end

list = ['X-Mailer']
data = Sisimai.make('/path/to/mbox', hook: callbackto, field: list)
json = Sisimai.dump('/path/to/mbox', hook: callbackto, field: list)

puts data[0].catch['x-mailer']      # Apple Mail (2.1283)

More information about the callback feature is available at Sisimai | How To Parse - Callback Page.

One-Liner

% ruby -rsisimai -e 'puts Sisimai.dump($*.shift)' /path/to/mbox

Output example

[{"action": "failed", "subject": "Nyaan", "catch": null, "token": "08acf78323edc7923a783c04749dd547ab45c433", "alias": "", "messageid": "[email protected]", "listid": "", "smtpcommand": "MAIL", "smtpagent": "Email::Sendmail", "lhost": "localhost", "timezoneoffset": "+0900", "feedbacktype": "", "senderdomain": "neko.example.jp", "diagnostictype": "SMTP", "softbounce": 1, "deliverystatus": "5.0.0", "addresser": "[email protected]", "diagnosticcode": "550 Unauthenticated senders not allowed", "timestamp": 1528523769, "destination": "example.com", "recipient": "[email protected]", "reason": "securityerror", "replycode": "550", "rhost": "neko.example.jp"}]

Sisimai Specification

Differences between Ruby version and Perl version

The following table show the differences between Ruby version of Sisimai and Perl version of Sisimai. Information about differences between Sisimai and bounceHammer are available at Sisimai | Differences page.

Features	Ruby version	Perl version
System requirements	Ruby 2.1 - 2.5	Perl 5.10 -
	JRuby 9.0.4.0-
Analytical precision ratio(2000 emails)[1]	1.00	1.00
The speed of parsing email(1000 emails)	3.75s[2]	1.70s
How to install	gem install	cpanm, cpm
Dependencies (Except core modules)	1 module	2 modules
LOC:Source lines of code	11700 lines	8500 lines
The number of tests(spec/,t/,xt/) directory	218000 tests	236000 tests
License	BSD 2-Clause	BSD 2-Clause
Support Contract provided by Developer	Coming soon	Available

1. See ./ANALYTICAL-PRECISION
2. Xeon E5-2640 2.5GHz x 2 cores | 5000 bogomips | 1GB RAM | Ruby 2.3.4

Other specification of Sisimai

• Parser Engines
• Bounce Reason List
• Data Structure of Sisimai::Data

Contributing

Bug report

Please use the issue tracker to report any bugs.

Emails could not be parsed

Bounce mails which could not be parsed by Sisimai are saved in the repository set-of-emails/to-be-debugged-because/sisimai-cannot-parse-yet. If you have found any bounce email cannot be parsed using Sisimai, please add the email into the directory and send Pull-Request to this repository.

Other Information

Related Sites

• @libsisimai | Sisimai on Twitter (@libsisimai)
• libSISIMAI.ORG | Sisimai | The successor to bounceHammer, Library to parse bounce mails
• Sisimai Blog | blog.libsisimai.org
• Facebook Page | facebook.com/libsisimai
• GitHub | github.com/sisimai/rb-Sisimai
• RubyGems.org | rubygems.org/gems/sisimai
• Perl verson | Perl version of Sisimai
• Fixtures | set-of-emails - Sample emails for "make test"

See also

• README-JA.md - README.md in Japanese(日本語)
• RFC3463 - Enhanced Mail System Status Codes
• RFC3464 - An Extensible Message Format for Delivery Status Notifications
• RFC3834 - Recommendations for Automatic Responses to Electronic Mail
• RFC5321 - Simple Mail Transfer Protocol
• RFC5322 - Internet Message Format

Author

@azumakuniyuki

Copyright

Copyright (C) 2015-2018 azumakuniyuki, All Rights Reserved.

License

This software is distributed under The BSD 2-Clause License.