BiotrackTHC Archives - Green Market Report

Dear Washington Cannabis Industry,

I write to you today with the sincere hope that I can cut through much of the noise and rumor of the last few days with some transparency so you can make informed decisions as business owners and so we can come together for the forward progress of the industry.

Before I write anything else, I want to emphasize the fact that Washington’s cannabis industry is incredibly important to BioTrack.  It was our first government contract.  TJ Ferraro – BioTrack’s founder – and I lived in Washington for the three months it took to customize and implement the original traceability system.  More licensees use our business platform in Washington than in any other state.  We have an office in Olympia, and nine BioTrack employees call Washington home.  Many of you are our friends.  If you take away nothing else from this letter, please know that you are important to us and we remain committed to doing everything in our power to make you successful.

Rather than ask you to blindly believe my narrative over someone else’s, the actual emails sent from me to the WSLCB are attached to the end of this letter so you can verify the facts and judge for yourself.  These emails are available via public records request so I am not sharing anything that wouldn’t already be available for public inspection.

What Is Going On?

BioTrackTHC’s traceability system contract with the WSLCB expires on October 31st, 2017 unless it is extended.  A more detailed chain of events is provided later, but the short story is that the WSLCB initially chose to not extend our contract beyond October 31, 2017, and MJ Freeway was selected to provide a replacement system that was to take over by the time our system is to be decommissioned, at midnight on October 31st.  It was recently announced by the WSLCB that the replacement system will not be operational in time, and licensees will have to report their seed-to-sale traceability data via manual spreadsheets for two months until the replacement system’s updated go-live date of January 2, 2018 assuming everything moving forward remains on schedule.  These manual spreadsheets are to be used for tracking all plant, harvest, inventory, conversion, sample, laboratory testing, transportation/chain-of-custody, and sales data for as long as the WSLCB’s contingency plan is in place.  

Is There Currently A Contract Extension On The Table Between The WSLCB And BioTrackTHC?

The WSLCB sent to BioTrack terms for an extension last Tuesday, October 17th.  This was the first offer for an extension that the WSLCB has offered BioTrack since MJ Freeway was awarded their contract in July, and remains as the current offer on the table.

The WSLCB offered BioTrack a four month extension for $125,000, or $31,250 per month.  

To put this offer in context, over the four-year life of the contract, BioTrack did not earn maintenance and support fees for the first two years and earned $180,000 per year ($15,000 per month) for the past two years.  The WSLCB’s extension offer is a $16,000 per month premium over the standard rate.

The WSLCB’s contract with the new vendor is $600,000 per year, or $50,000 per month.

Per my email to the WSLCB on Thursday, October 19th(see “Exhibit 2”), BioTrack did NOT decline the WSLCB’s offer for extension.  However, BioTrack requires resolution on security concerns that were previously brought to the WSLCB’s attention before the other components of any offer, such as financial and timing components, can even be considered.  BioTrack is still actively seeking to resolve these security concerns prior to the expiration of the contract.

The final paragraph in my last email to the WSLCB on this matter, dated October 19th, reads as follows:

“When we first spoke last Monday about the possibility of extension, you assured me multiple times that the current project is running on schedule and that the extension was being offered to us for the benefit of the third-party software providers, and for that we are grateful for the WSLCB’s consideration.  I want to be clear that we are not saying “no” to the extension.  We just cannot consider any other factors until we can resolve these concerns and they have not yet been resolved.  However, we don’t want our security concerns to cause a burden on the WSLCB if everything is indeed running on time.  We have done our best to be partners with the WSLCB since the beginning so we hope that we can resolve our concerns before next week’s transition.”

What Is The Security Concern That Needs To Be Resolved Before BioTrack Can Feel Comfortable In Accepting The Extension?

On Monday October 9th, the WSLCB and I connected for the first time in nearly six months to discuss a possible extension.  I was informed that the WSLCB remained confident that the new system was on time, but that an extension would allow the business seed-to-sale software providers more time to integrate with the new government system.  I then informed the WSLCB that BioTrack has serious concerns related to security.  

After MJ Freeway was awarded the contract, beginning the week of August 21, 2017, BioTrack began providing a “data dump” of the entire traceability system database to the WSLCB on a weekly basis so that the entire dataset could be mapped and migrated to the replacement system. However, many Washington licensees received an email in mid-September alleging to sell databases described as “WA DATABASE,” “NV PROD DATABASE,” and “PA PROD DATABASE,” among others (see “Exhibit 6”).  These presumably are to mean the Washington database, the Nevada database, and the Pennsylvania database.  The emails also provided unencrypted sample data files as a kind of “proof of life.”  Some business seed-to-sale software providers took it upon themselves to investigate the sample data and it was reported that the sample data not only appeared legitimate, but that it included sensitive data that is not publicly available: data that is contained within the full un-redacted traceability dataset.  I am sure that many of my peers contributed to the industry’s investigation, but I specifically want to recognize David Busby, CEO of WeedTraQR, for his tireless efforts in this regard.

To BioTrack, other third-party software providers, and many Washington licensees, this is a serious concern.  BioTrack currently operates six state-level government cannabis traceability systems and has managed Washington’s traceability system for four years without any security breaches.  We then find ourselves in a situation where both our reputation and our security are co-mingled with another company’s; and then a few months later, credible reports surface that Washington-specific data not otherwise available to the public is found outside of the chain of custody.  

I conveyed to the WSLCB our concern that this situation where we “share space” with their new vendor puts us in jeopardy.  I memorialized those concerns in writing within my follow-up email dated October 16th (see “Exhibit 1”).

“The current status quo has already harmed both our reputation and our peace of mind with respect to security risk.  Please understand that we have continued to provide our traceability technology, support, and weekly data dumps of the entirety of the database because we remain contractually obligated to do so, not because the new status quo is in any way comfortable for us.  Every passing day in which we find our reputation and security co-mingled with another vendor without any assurances that our technology—and therefore our livelihoods—are safe within this new co-mingled environment compounds our anxiety and intensifies our desire to exit the unsafe situation.”

Our technology is how we make our living.  If the security of our technology becomes compromised, at least sixty people lose their jobs and all of our customers who depend on us also become compromised.  It would be irresponsible of us to ignore credible threats to technology security.

Now I am not saying that we know for certain that the WSLCB’s or MJ Freeway’s security was breached.  Maybe there was no security breach of any kind.  Maybe there was a security breach and it has since been remedied.  Maybe there was a security breach and it’s still there.  What we do know is that there is enough smoke that we are not comfortable moving forward without a reasonable level of assurance that the fire has been addressed.

The WSLCB’s position at the time was that the email was a “spoof” and that it was “fake news”, and BioTrack respects their prerogative to believe that no security issues exist.  However, we respectfully disagreed with that position and said we needed some type of meaningful assurances that the alleged breach either did not happen or did happen and has since been remedied, since without that we have no solid footing in understanding our current risk exposure.

What Has BioTrack Obtained So Far To Address The Security Concerns?

To emphasize BioTrack’s sincere interest in a possible extension and to ensure that BioTrack’s concern regarding the possible security issue was not misunderstood, I had a member of BioTrack’s board of directors join me on a call with the WSLCB on Friday, October 13th.  I also had the CEO of one of our competitors join the call to show what he had uncovered from the “spoof” email that licensees had received.  At one point, we suggested that a third-party security audit providing a “clean bill of health” may go a long way in allaying our concerns.  The WSLCB reassured us that a security audit had been performed by the Washington State Office of the Chief Information Officer (OCIO), but that none of the contents of the audit report could be provided to us.  The WSLCB offered to obtain a statement from the OCIO that could address our concerns.  We agreed to incorporate it into our overall evaluation, but could not promise that it would allay our concerns since we had not yet seen it.

To expedite the process in good faith, immediately after that call concluded, BioTrack submitted a records request to the OCIO for any security audit documentation that is available to the public.  As of the writing of this letter we have received one response from the OCIO’s office dated October 19th stating that they estimate, “it will require no more than thirty days to provide you a response,” (see “Exhibit 3”).

On October 17th, I received an email from the WSLCB stating that BioTrack’s “concerns were addressed already.”  Up to that point, we had received only verbal assertions and nothing in writing.  One part of my October 19th email (see “Exhibit 2”) contained the following response:

“We appreciate the fact that the WSLCB is leaning on a review performed by the OCIO that found no adverse security concerns, but we have not seen any documentation with our own eyes or even a document stating that we are prohibited from seeing such documentation.  No offense, but we cannot just take your verbal word on something that could have far reaching consequences for our livelihoods and our customers. 

I am an accountant by training.  If someone withdraws $1,000 from the company bank account, that person would have to show me a receipt proving where it went.  A response of, “I have the receipt, but I cannot produce it for you,” is not one than anyone could reasonably accept.    

I’m not trying to make light of the situation, but please appreciate the position we’re in in that nothing that we can rely on has been provided to us.”

Later that day, the WSLCB sent to me a letter from the State of Washington Office of Cyber Security (see “Exhibit 4”).  The letter states, “We have completed our security design review on the new cannabis traceability system provided by Leaf Data Systems vendor MJ Freeway… the project, as proposed, uses appropriate security controls and methods to meet OCIO IT security standards at the time of review.”  Though we greatly appreciate the efforts of the WSLCB staff to obtain this letter, it does not provide much information or the peace of mind that we are seeking.  

1. The letter does not provide a date for when the review was performed.  No review can provide any assurances about current system security if it was performed before the “spoof” email was sent to licensees.  
2. The letter states that it was the security design that was reviewed; the system itself did not undergo generally accepted security audit testing.  That is like the difference between, “Patrick, we reviewed the airplane design you drew on paper and the design should fly,” versus, “Patrick we tested the actual plane you built and it successfully flew.”
3. Finally, the OCIO’s online project dashboard’s OCIO Assessment reads, “user authentication requirements and Security Design Review increase risk due to imminent project implementation deadline” (emphasis mine) for 07/13/2017, 07/27/2017, 08/22/017, and 09/15/2017.  Now, I don’t know what this means, and to be fair I only just found this today and have not given the WSLCB an opportunity to help me understand what this comment means, but my current interpretation is that on each of those dates, the Security Design Review continued to increase the risk that the project would not be completed on time because the review remained ongoing as the system was being developed (see “Exhibit 5”).

(https://http://waocio.force.com/ProjectDetail?id=a060P00000ezEk1QAE)

The WSLCB has been quoted recently in the media saying, “We’ve given them everything that we have and every assurance.”  Now I understand that this is likely true; that the WSLCB has given to us what they are allowed to give us.  However, everything they have given us thus far has been verbal and one brief letter on which we cannot place a great deal of reliance.  Again, we have a responsibility to our other government clients, to the licensees who depend on our business software, and our staff to take every reasonable precaution to protect our technology from security risks.  Accepting any extension of the current situation without reasonable assurances, regardless of the amount of money offered, would be irresponsible.

Again, we did not decline the WSLCB’s extension request.  We just cannot move forward until these concerns are dealt with.  We are still actively searching for alternative means to help us determine how sensitive non-public data came to be found within the “spoof” email sent to the industry and welcome any assistance from any other party, WSLCB or otherwise.

Can BioTrack Accept The Extension After October 31st If The Security Issue Is Addressed Shortly After?

I am not sure as I am not an attorney that specializes in Washington’s government contract law.  However, I do not believe either the WSLCB or BioTrack can “extend” a contract that is no longer in effect.  There may be a way to justify a sole-source procurement where the WSLCB can offer a new contract should the current contract expire, but we would have to consult an attorney.

What Is BioTrack’s Plan If The Contract Expires on October 31st?

We learned about the WSLCB’s “contingency plan” from the same announcement that many others in the industry received on Thursday, October 19th, and we learned on Tuesday October 24th, with everyone else that the manual spreadsheet era is expected to last at least through January 1st… so many of our plans are rapidly evolving and still solidifying.  That being said, here is our game plan for now.

First, BioTrack is committed to its direct commercial customers: those who rely on BioTrack’s business system for inventory management and point-of-sale.  It is our intent that in every way possible, your BioTrack business system will automatically generate the spreadsheets necessary for submission to the WSLCB so that there is a reduced impact to your business.  Please have patience with us as we are working with a moving target.

Secondly, the success of Washington’s industry as a whole – and therefore the success of every licensee in Washington whether you use our business platform or not — is important to us.  We have no intention of giving the federal government any reason to give this industry a hard time.  BioTrack understands that even with manual spreadsheets, there needs to be some method of communication and data exchange between licensees regardless of which third-party commercial system you use.  One common denominator for every third-party commercial software system in Washington is that it successfully integrates with our API.  Because BioTrack owns its traceability technology and licenses it to state governments for use, we can create a private-sector version of our traceability system that would mirror the current traceability system.  It would even include a web-interface for the licensees who have relied on the freely-provided MJ Traceability website, and it would have the current version of the Washington API so every current business seed-to-sale provider will already be integrated with it.  Though we are still working on the specific mechanics, all it would take is for everyone to point their systems to the new URL (website); all functions and all data that is currently coordinated and exchanged between licensees would be nearly identical, if not perfectly identical, to the way things presently work.  This private-sector “clone” of BioTrack’s traceability system could continue to operate for as long as we need it to, even if a worst-case scenario were to happen and the WSLCB’s system is unable to go-live by January 1, 2018 as planned.  We have yet to figure out the economics, but our goal is to just get the job done first and worry about the rest later.

Please remember that we are attempting to surf a wave in the wild here, so I can guarantee you that there will be turbulence as we go; however, my team and I believe that this is our best option to avoid industry Armageddon and we will all band together to navigate these unpredictable waters as best we can.  We have already received an outpouring of support from the other third-party software systems and in spite of the fact that we’re competitors and have our differences, I know that we can continue to use this challenge as an opportunity to bring the industry together for everyone’s success.

 

Highest Regards,

Patrick Vo

President and CEO

BioTrackTHC

 

Exhibit 1

Email — October 16, 2017

No changes have been made to this email reproduction other than the removal of recipients who were cc’d.

—    —

From: Patrick Vo <patrick.vo@biotrackthc.com>
Date: Mon, Oct 16, 2017 at 12:28 PM
Subject: BioTrack – Follow-up On Friday’s Call
To: “Antolin, Peter P (LCB)” <peter.antolin@lcb.wa.gov>

Peter,

First of all, because the question came up on Friday’s call, I want to reiterate that we are indeed interested in the possibility of an extension.  Neither I, nor Director Molloy would have invested our time for the call were that not the case.  However, though it is the LCB’s prerogative to believe that there are no security issues related to the LCB or any of the LCB’s other vendors, we’ve heard otherwise from other third-party vendors within the state who have uncovered concerning evidence to the contrary.

It was also asked of us why security is an issue when all the LCB is asking for is an extension of the status quo.  To reiterate and clarify what Mr. Molloy stated on the call, that question assumes that we are okay with the status quo.  However, the status quo changed when another company was chosen to essentially be a co-vendor with the LCB and BioTrack, followed by reports surfacing that Washington-specific data not otherwise available to the public began being distributed.   

The current status quo has already harmed both our reputation and our peace of mind with respect to security risk.  Please understand that we have continued to provide our traceability technology, support, and weekly data dumps of the entirety of the database because we remain contractually obligated to do so, not because the new status quo is in any way comfortable for us.  Every passing day in which we find our reputation and security co-mingled with another vendor without any assurances that our technology—and therefore our livelihoods—are safe within this new co-mingled environment compounds our anxiety and intensifies our desire to exit the unsafe situation. 

With respect to reputational harm, at least two state agencies have reached out to one of our other government clients with concerns that they heard that the “Washington Traceability System” was allegedly compromised.  At this point, BioTrack has become synonymous with Washington’s Traceablity System and so we had to assertively defend ourselves; and these are just the two inquiries that we know about.  This cannot continue.

With respect to security risk, because LCB owns the data captured by the traceablity system (including password hashes included within the database), we currently have no viable method available to us of ensuring perfect security of our systems within the current co-mingled situation.  Even if every single password was changed tomorrow, this would only provide temporary relief until the next weekly file. 

Thankfully, the third-party vendors who have been independently investigating this matter have yet to uncover evidence that passwords themselves have been compromised.  Nevertheless, their findings thus far suggest that there is indeed a breach somewhere. 

On the call, we asked for assurances that the alleged breach either did not happen or has since been remedied if it did happen.  Please remember that we at BioTrack do not know what the LCB knows, and so we have no solid footing when it comes to understanding our risk exposure to being co-vendors with the LCB’s other system provider.  We are not asking for anything inappropriate.  We expect the LCB to protect their own proprietary information, and that of any other vendor, just as we would expect the LCB to protect our own proprietary information.  However, we don’t know what to ask for because we don’t know what we don’t know. 

The consensus from the majority of our board of directors is that, though the LCB’s request for an extension remains on the table, the current situation represents an imminent and unknown risk.  We have a responsibility to our customers, employees, and shareholders to not endanger our own security. 

You mentioned on the call that the vendor with whom the reported evidence appears to implicate has already completed a security audit or something similar by the OCIO.  In order to keep the momentum of our conversation moving as quickly as possible, and to avoid putting the agency in an awkward position, we submitted a public records request on Friday for a (presumably redacted) copy of the completed information technology security audit for that vendor.  We know that it may take a few days to receive whatever redacted information will be made available for public inspection, but whatever is contained therein might go a long way towards allaying our concerns. 

Finally, you requested a dollar figure from us earlier in the week, and once again Friday.  We have made good faith attempts at determining a price, but without knowing the risk factor we cannot put a price on the unknown potential exposure component.  The value of the contract to BioTrack is its revenue less its costs; and the probability of costs related to potential events such as a security breach, our source code made public, reputation loss, commercial client loss, other government client loss, litigation, etc… are undefinably higher now than they were prior to the other vendor’s co-existence with BioTrack and the LCB.  Better understanding those risks are a prerequisite to determining the likely costs and therefore the contract extension as a whole.   

I know that this is not the answer you were hoping for; but it is the most responsible answer I can give you given the circumstances and the concerns of our directors.  We will wait for the public records from OCIO and will continue to work with other third-party vendors to independently investigate the reported matters in order to reach a conclusion with respect to our risk exposure moving forward and that risk exposure’s appropriate price. 

We understand that the LCB is pressed for time and may need to withdraw its request for an extension if you are not able to wait for us to receive and inspect the requested public documents from OCIO.  Any further assistance the LCB can provide in allaying our concerns and the concerns of other third-party vendors would be greatly appreciated.  We hope to come to a resolution in the best interest of all parties that does not sacrifice security, and I trust that you would expect nothing less.

Sincerely,

Exhibit 2

Email — October 19, 2017

No changes have been made to this email reproduction other than the removal of recipients who were cc’d.

—    —

From: Patrick Vo <patrick.vo@biotrackthc.com>
Date: Thu, Oct 19, 2017 at 12:21 PM
Subject: Re: BioTrack – Follow-up On Friday’s Call
To: “Antolin, Peter P (LCB)” <peter.antolin@lcb.wa.gov>

Peter,

I know I sound like a broken record, but I registered our concerns again because we do not feel that those concerns have been sufficiently addressed.  I am not sure what you meant by “the most we could offer was the information that we provided last week,” as we were not provided anything of substance.  We appreciate the fact that the LCB is leaning on a review performed by the OCIO that found no adverse security concerns, but we have not seen any documentation with our own eyes or even a document stating that we are prohibited from seeing such documentation.  No offense, but we cannot just take your verbal word on something that could have far reaching consequences for our livelihoods and our customers. 

I am an accountant by training.  If someone withdraws $1,000 from the company bank account, that person would have to show me a receipt proving where it went.  A response of, “I have the receipt, but I cannot produce it for you,” is not one than anyone could reasonably accept. 

Here is how a hypothetical conversation with my board of directors or one of our customers may go with what we have been provided thus far:

      Patrick:    “The LCB tells me that everything is fine and that we should not be concerned.”

      Board:     “How do you know?”

      Patrick:    “Because they told me so.”

      Board:     “Have you reviewed the evidence that supports this conclusion?”

      Patrick:    “No.  The LCB tells me that I am prohibited from reviewing the evidence.”

      Board:     “How do you know you cannot review the evidence?”

      Patrick:    “Because they told me so.”

 

I’m not trying to make light of the situation, but please appreciate the position we’re in in that nothing that we can rely on has been provided to us.

Thank you for taking a second look at the spoof email.  Other third-party software providers have analyzed the sample database provided from that spoof email and have reported that it contained the following information that is not available in the FOIA data that the LCB releases:

• User email addresses of vendor staff and LCB staff,
• Licensee employees, drivers, and vehicles (tag and VIN).

Obviously, there’s not much you can do regarding the Nevada data and the Pennsylvania data, but any insight you could provide on how this information left LCB custody would be helpful.

I understand that all Washington agencies are held to the same public disclosure standards, whether LCB, OCIO, or otherwise.  However, as stated earlier, even a reply from OCIO stating that the document exists but cannot be made available for public inspection is still more than what I have at the moment.  We are doing everything we can to quickly get comfort over the discomforting red flags and are leaving no stone unturned. 

The potential security risks are a major concern, but I would be remiss to not also reiterate the reputational damage as a concern.  Whether or not the LCB believes that the alleged security breaches are real, state agencies from across the country have heard about and expressed concern regarding them.  We remain at the forefront of having to defend both our credibility and the LCB’s credibility to current and prospective government clients without any assurance that we will not have to continue to do so should we agree an extension

When we first spoke last Monday about the possibility of extension, you assured me multiple times that the current project is running on schedule and that the extension was being offered to us for the benefit of the third-party software providers, and for that we are grateful for the LCB’s consideration.  I want to be clear that we are not saying “no” to the extension.  We just cannot consider any other factors until we can resolve these concerns and they have not yet been resolved.  However, we don’t want our security concerns to cause a burden on the LCB if everything is indeed running on time.  We have done our best to be partners with the LCB since the beginning so we hope that we can resolve our concerns before next week’s transition. 

__________

For the copies of the emails:

https://goo.gl/4rhvZa