Add a certificate store

Add a certificate store

[ekasper, 2013-07-25 11:17:39]

This is preparation work for scanning the log with get-entries and storing the
results.

It seemed reasonable to make the cert store a reusable standalone component.
Monitors can then store monitoring metadata in their custom data stores, with
sha256 references to the cert store.

I also added a temporary index-by-domain to the database so that I can set up a
demo dashboard search and get on with life. I'll move this out of the cert store
and into monitoring later on.

This CL also refactors common parts of sqlite connection management into
sqlite_connection.py.

[Ben Laurie (Google), 2013-07-25 16:49:16]

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/cert_d...
File src/python/ct/client/cert_db_test.py (right):

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/cert_d...
src/python/ct/client/cert_db_test.py:39: self.assertEqual("hello\x00", cert)
I don't understand how this shows the db ignores duplicates?

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
File src/python/ct/client/sqlite_cert_db.py (right):

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_cert_db.py:38: def __process_name(subject,
reverse=True):
This belongs in the parent.

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_cert_db.py:71:
(sqlite3.Binary(hashlib.sha256(der_cert).digest()),
Seems to me that the hash is common to all db backends and should be calculated
by the parent.

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
File src/python/ct/client/sqlite_log_db.py (right):

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_log_db.py:39: def __encode_log_metadata(self,
metadata):
Should be in parent?

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_log_db.py:46: def __decode_log_metadata(self,
log_server, serialized_metadata):
Should be in parent?

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_log_db.py:86: def __encode_sth(self, audited_sth):
Should be in parent?

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_log_db.py:96: def __decode_sth(self, sth_row):
Should be in parent?

[ekasper, 2013-07-25 17:19:57]

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/cert_d...
File src/python/ct/client/cert_db_test.py (right):

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/cert_d...
src/python/ct/client/cert_db_test.py:39: self.assertEqual("hello\x00", cert)
On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
> I don't understand how this shows the db ignores duplicates?

Fixed.

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
File src/python/ct/client/sqlite_cert_db.py (right):

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_cert_db.py:38: def __process_name(subject,
reverse=True):
On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
> This belongs in the parent.

Not really.

It probably belongs somewhere in x509_name - I've added a TODO but I'll leave it
here until I get round to implementing the SAN extension, okay?

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_cert_db.py:71:
(sqlite3.Binary(hashlib.sha256(der_cert).digest()),
On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
> Seems to me that the hash is common to all db backends and should be
calculated
> by the parent.

Agreed, done.

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
File src/python/ct/client/sqlite_log_db.py (right):

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_log_db.py:39: def __encode_log_metadata(self,
metadata):
On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
> Should be in parent?

Nope, it's specific to the sqlite schema.

(Also, this is old code.)

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_log_db.py:46: def __decode_log_metadata(self,
log_server, serialized_metadata):
On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
> Should be in parent?

Ditto.

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_log_db.py:86: def __encode_sth(self, audited_sth):
On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
> Should be in parent?

Ditto.

https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite...
src/python/ct/client/sqlite_log_db.py:96: def __decode_sth(self, sth_row):
On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
> Should be in parent?

Ditto.

[ekasper, 2013-07-30 09:49:03]

Ping.


On Thu, Jul 25, 2013 at 7:19 PM, <ekasper@google.com> wrote:

>
> https://codereview.appspot.**com/11828043/diff/5001/src/**
>
python/ct/client/cert_db_test.**py<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/cert_db_test.py>
> File src/python/ct/client/cert_db_**test.py (right):
>
> https://codereview.appspot.**com/11828043/diff/5001/src/**
>
python/ct/client/cert_db_test.**py#newcode39<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/cert_db_test.py#newcode39>
> src/python/ct/client/cert_db_**test.py:39: self.assertEqual("hello\x00",
> cert)
> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>
>> I don't understand how this shows the db ignores duplicates?
>>
>
> Fixed.
>
>
> https://codereview.appspot.**com/11828043/diff/5001/src/**
>
python/ct/client/sqlite_cert_**db.py<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_cert_db.py>
> File src/python/ct/client/sqlite_**cert_db.py (right):
>
> https://codereview.appspot.**com/11828043/diff/5001/src/**
>
python/ct/client/sqlite_cert_**db.py#newcode38<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_cert_db.py#newcode38>
> src/python/ct/client/sqlite_**cert_db.py:38: def __process_name(subject,
> reverse=True):
> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>
>> This belongs in the parent.
>>
>
> Not really.
>
> It probably belongs somewhere in x509_name - I've added a TODO but I'll
> leave it here until I get round to implementing the SAN extension, okay?
>
>
> https://codereview.appspot.**com/11828043/diff/5001/src/**
>
python/ct/client/sqlite_cert_**db.py#newcode71<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_cert_db.py#newcode71>
> src/python/ct/client/sqlite_**cert_db.py:71:
> (sqlite3.Binary(hashlib.**sha256(der_cert).digest()),
> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>
>> Seems to me that the hash is common to all db backends and should be
>>
> calculated
>
>> by the parent.
>>
>
> Agreed, done.
>
>
> https://codereview.appspot.**com/11828043/diff/5001/src/**
>
python/ct/client/sqlite_log_**db.py<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_log_db.py>
> File src/python/ct/client/sqlite_**log_db.py (right):
>
> https://codereview.appspot.**com/11828043/diff/5001/src/**
>
python/ct/client/sqlite_log_**db.py#newcode39<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_log_db.py#newcode39>
> src/python/ct/client/sqlite_**log_db.py:39: def
> __encode_log_metadata(self, metadata):
> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>
>> Should be in parent?
>>
>
> Nope, it's specific to the sqlite schema.
>
> (Also, this is old code.)
>
>
> https://codereview.appspot.**com/11828043/diff/5001/src/**
>
python/ct/client/sqlite_log_**db.py#newcode46<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_log_db.py#newcode46>
> src/python/ct/client/sqlite_**log_db.py:46: def
> __decode_log_metadata(self, log_server, serialized_metadata):
> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>
>> Should be in parent?
>>
>
> Ditto.
>
>
> https://codereview.appspot.**com/11828043/diff/5001/src/**
>
python/ct/client/sqlite_log_**db.py#newcode86<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_log_db.py#newcode86>
> src/python/ct/client/sqlite_**log_db.py:86: def __encode_sth(self,
> audited_sth):
> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>
>> Should be in parent?
>>
>
> Ditto.
>
>
> https://codereview.appspot.**com/11828043/diff/5001/src/**
>
python/ct/client/sqlite_log_**db.py#newcode96<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_log_db.py#newcode96>
> src/python/ct/client/sqlite_**log_db.py:96: def __decode_sth(self,
> sth_row):
> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>
>> Should be in parent?
>>
>
> Ditto.
>
>
https://codereview.appspot.**com/11828043/<https://codereview.appspot.com/118...
>

[Ben Laurie (Google), 2013-07-30 09:51:42]

Sorry, on it now.


On 30 July 2013 10:49, Emilia Kasper <ekasper@google.com> wrote:

> Ping.
>
>
> On Thu, Jul 25, 2013 at 7:19 PM, <ekasper@google.com> wrote:
>
>>
>> https://codereview.appspot.**com/11828043/diff/5001/src/**
>>
python/ct/client/cert_db_test.**py<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/cert_db_test.py>
>> File src/python/ct/client/cert_db_**test.py (right):
>>
>> https://codereview.appspot.**com/11828043/diff/5001/src/**
>>
python/ct/client/cert_db_test.**py#newcode39<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/cert_db_test.py#newcode39>
>> src/python/ct/client/cert_db_**test.py:39: self.assertEqual("hello\x00",
>> cert)
>> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>>
>>> I don't understand how this shows the db ignores duplicates?
>>>
>>
>> Fixed.
>>
>>
>> https://codereview.appspot.**com/11828043/diff/5001/src/**
>>
python/ct/client/sqlite_cert_**db.py<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_cert_db.py>
>> File src/python/ct/client/sqlite_**cert_db.py (right):
>>
>> https://codereview.appspot.**com/11828043/diff/5001/src/**
>>
python/ct/client/sqlite_cert_**db.py#newcode38<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_cert_db.py#newcode38>
>> src/python/ct/client/sqlite_**cert_db.py:38: def __process_name(subject,
>> reverse=True):
>> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>>
>>> This belongs in the parent.
>>>
>>
>> Not really.
>>
>> It probably belongs somewhere in x509_name - I've added a TODO but I'll
>> leave it here until I get round to implementing the SAN extension, okay?
>>
>>
>> https://codereview.appspot.**com/11828043/diff/5001/src/**
>>
python/ct/client/sqlite_cert_**db.py#newcode71<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_cert_db.py#newcode71>
>> src/python/ct/client/sqlite_**cert_db.py:71:
>> (sqlite3.Binary(hashlib.**sha256(der_cert).digest()),
>> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>>
>>> Seems to me that the hash is common to all db backends and should be
>>>
>> calculated
>>
>>> by the parent.
>>>
>>
>> Agreed, done.
>>
>>
>> https://codereview.appspot.**com/11828043/diff/5001/src/**
>>
python/ct/client/sqlite_log_**db.py<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_log_db.py>
>> File src/python/ct/client/sqlite_**log_db.py (right):
>>
>> https://codereview.appspot.**com/11828043/diff/5001/src/**
>>
python/ct/client/sqlite_log_**db.py#newcode39<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_log_db.py#newcode39>
>> src/python/ct/client/sqlite_**log_db.py:39: def
>> __encode_log_metadata(self, metadata):
>> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>>
>>> Should be in parent?
>>>
>>
>> Nope, it's specific to the sqlite schema.
>>
>> (Also, this is old code.)
>>
>>
>> https://codereview.appspot.**com/11828043/diff/5001/src/**
>>
python/ct/client/sqlite_log_**db.py#newcode46<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_log_db.py#newcode46>
>> src/python/ct/client/sqlite_**log_db.py:46: def
>> __decode_log_metadata(self, log_server, serialized_metadata):
>> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>>
>>> Should be in parent?
>>>
>>
>> Ditto.
>>
>>
>> https://codereview.appspot.**com/11828043/diff/5001/src/**
>>
python/ct/client/sqlite_log_**db.py#newcode86<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_log_db.py#newcode86>
>> src/python/ct/client/sqlite_**log_db.py:86: def __encode_sth(self,
>> audited_sth):
>> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>>
>>> Should be in parent?
>>>
>>
>> Ditto.
>>
>>
>> https://codereview.appspot.**com/11828043/diff/5001/src/**
>>
python/ct/client/sqlite_log_**db.py#newcode96<https://codereview.appspot.com/11828043/diff/5001/src/python/ct/client/sqlite_log_db.py#newcode96>
>> src/python/ct/client/sqlite_**log_db.py:96: def __decode_sth(self,
>> sth_row):
>> On 2013/07/25 16:49:16, Ben Laurie (Google) wrote:
>>
>>> Should be in parent?
>>>
>>
>> Ditto.
>>
>>
https://codereview.appspot.**com/11828043/<https://codereview.appspot.com/118...
>>
>
>

[Ben Laurie (Google), 2013-07-30 09:53:26]

LGTM