Greetings,

News articles say FBI is probably going to scrap its $170 Million
Virtual Case File system and start over from scratch.

The problem is characterized as a software issue. Not being an
enterprise architect, I would have thought the problem was an enterprise
architecture issue: hardware, software, and mostly organizational wetware.

I'm posting here because of the number of posters and lurkers with
significant architectural experience in mainframe and enterprise systems.

One article says the FBI wants to do prognostic data mining. Who
doesn't? For $170 Million, the FBI is going to invent a new field (for
which there surely must be players on Wall Street who would pay ten
times as much or more for the same capability), replace hardware, create
a nationwide secure network that can handle the most sensitive national
security information _and_ be accessible to garden-variety FBI agents,
_and_ that said agents will be willing to use. Even if they didn't have
to upgrade a system currently in use (which is what Sen. Patrick Leahy
has said is the problem), it sounds like somebody slipped a decimal
point (at least) in time and money.

This is the future of computing as visualized from when spinning tape
drives were the very symbol of the awesome capacity of computers to
acquire and to sort through information: a computer that gobbles up
everything, sifts through it all, and tells the hapless human what to do
next. Still science fiction.

Projects that come to mind that have to work and that historically have
worked are all mainframe-type systems, some of which I think are still
using System 360-vintage software. You don't invent the future all at
once, you do it piece by piece, and you probably hire a mainframe house
to do it for you over a period of decades. SAIC couldn't do it for $170
Million in a few years? Gimme a break.

If space-type programs didn't have such a lousy track record (the
Manhattan and Apollo projects worked, the Space Shuttle, the War on
Cancer, and Japan's Fifth Generation AI Project didn't), I'd say that a
space-type program was what would really be required, with that kind of
visibility and commitment. Won't happen, whether it would be a good
idea or not, because time and the tides have moved past computing for
that sort of attention.

Maybe they should hire Dell and an offshore software house--the
combination that has set the expectation for the price tag and for what
constitutes a working computer architecture.

RM

From:	Del Cecchi
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Fri, 14 Jan 2005 10:58:06 -0600

Robert Myers wrote:
> Greetings,
>
> News articles say FBI is probably going to scrap its $170 Million
> Virtual Case File system and start over from scratch.
>
> The problem is characterized as a software issue. Not being an
> enterprise architect, I would have thought the problem was an enterprise
> architecture issue: hardware, software, and mostly organizational wetware.
>
> I'm posting here because of the number of posters and lurkers with
> significant architectural experience in mainframe and enterprise systems.
>
> One article says the FBI wants to do prognostic data mining. Who
> doesn't? For $170 Million, the FBI is going to invent a new field (for
> which there surely must be players on Wall Street who would pay ten
> times as much or more for the same capability), replace hardware, create
> a nationwide secure network that can handle the most sensitive national
> security information _and_ be accessible to garden-variety FBI agents,
> _and_ that said agents will be willing to use. Even if they didn't have
> to upgrade a system currently in use (which is what Sen. Patrick Leahy
> has said is the problem), it sounds like somebody slipped a decimal
> point (at least) in time and money.
>
> This is the future of computing as visualized from when spinning tape
> drives were the very symbol of the awesome capacity of computers to
> acquire and to sort through information: a computer that gobbles up
> everything, sifts through it all, and tells the hapless human what to do
> next. Still science fiction.
>
> Projects that come to mind that have to work and that historically have
> worked are all mainframe-type systems, some of which I think are still
> using System 360-vintage software. You don't invent the future all at
> once, you do it piece by piece, and you probably hire a mainframe house
> to do it for you over a period of decades. SAIC couldn't do it for $170
> Million in a few years? Gimme a break.
>
> If space-type programs didn't have such a lousy track record (the
> Manhattan and Apollo projects worked, the Space Shuttle, the War on
> Cancer, and Japan's Fifth Generation AI Project didn't), I'd say that a
> space-type program was what would really be required, with that kind of
> visibility and commitment. Won't happen, whether it would be a good
> idea or not, because time and the tides have moved past computing for
> that sort of attention.
>
> Maybe they should hire Dell and an offshore software house--the
> combination that has set the expectation for the price tag and for what
> constitutes a working computer architecture.
>
> RM
>
How about a nice Blue Gene based system like the Mayo Clinic is building
for genomic medicine and mining patient records? :-)

Maybe they shouldn't have hired the low bidder. And the government gets
what they ask for in the contract. Maybe they asked for the wrong
thing, or maybe they asked for the undoable.

Really wants one want to have them take over health care, eh?

del cecchi

From:	Bernd Paysan
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Fri, 14 Jan 2005 23:38:56 +0100

Del Cecchi wrote:
> Maybe they shouldn't have hired the low bidder. And the government gets
> what they ask for in the contract. Maybe they asked for the wrong
> thing, or maybe they asked for the undoable.

IMHO, if you are asking for new technology, you should not make a bit, but a
contest. New ideas mean that many people go and head off into the wrong
direction. There's nothing wrong with that, you can stop them pretty soon
(maybe even before they get into the Death Valley of the project).
"Contest" means that you allow all participants to deliver a prototype -
but the price you'll pay is fixed, and unevenly distributed among the
participants (the higher the rank, the more).

I think all the basic components for such a system are already available.
All they need for a start is a hypertext (wiki) system where officers can
enter data, add links and google around (not all free-form, but a
significant part of the case file is like that). A thesaurus should help
officers to enter precise worded entries, but I think that is common
practice for real files, too (the thesaurus should be built into the search
matrix too, e.g. if you search for "body", you should find "corpse", too -
Google doesn't). Biometric data can be automatically put into categories,
too, e.g. run fingerprint and face recognition software to identify
potentially overlapping cases.

Perhaps the FBI aimed much too high - that is the most common problem with
government contracts. They rarely take the approach "start low, with what's
available, and improve gradually", they always want the big thing, and are
willing to wait ten years and hundred millions, and then scrap the whole
thing.

The harder thing is to prevent abuse of such a system. The easy part is to
expire and clear files, though probably nothing will ever get deleted (at
least searching should require a sort of "way back machine", i.e. not in
the normal context). However, criminal officers can use such a virtual file
and the search functions to hide. Therefore, each file should also keep
track of who viewed it - if an officer not in charge views a particular
file quite often, there's a suspect. Version control is also obvious. That
all this should happen within a VPN, too.

The web is a huge public general purpose information system, making a
specialized in-house variant should really not take that much. The search
engine companies sell their technology, they'll probably be happy with
ideas how to make things better (a real-world Google could benefit from a
thesaurus, too).

--
Bernd Paysan
"If you want it done right, you have to do it yourself"
http://www.jwdt.com/~paysan/

From:	Robert Myers
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Fri, 14 Jan 2005 18:13:34 -0500

On Fri, 14 Jan 2005 23:38:56 +0100, Bernd Paysan
wrote:

>Del Cecchi wrote:
>> Maybe they shouldn't have hired the low bidder. And the government gets
>> what they ask for in the contract. Maybe they asked for the wrong
>> thing, or maybe they asked for the undoable.
>
>IMHO, if you are asking for new technology, you should not make a bit, but a
>contest. New ideas mean that many people go and head off into the wrong
>direction. There's nothing wrong with that, you can stop them pretty soon
>(maybe even before they get into the Death Valley of the project).
>"Contest" means that you allow all participants to deliver a prototype -
>but the price you'll pay is fixed, and unevenly distributed among the
>participants (the higher the rank, the more).
>

Hasn't gotten DARPA an autonomous land vehicle, so far.

>I think all the basic components for such a system are already available.
>All they need for a start is a hypertext (wiki) system where officers can
>enter data, add links and google around (not all free-form, but a
>significant part of the case file is like that). A thesaurus should help
>officers to enter precise worded entries, but I think that is common
>practice for real files, too (the thesaurus should be built into the search
>matrix too, e.g. if you search for "body", you should find "corpse", too -
>Google doesn't). Biometric data can be automatically put into categories,
>too, e.g. run fingerprint and face recognition software to identify
>potentially overlapping cases.
>
>Perhaps the FBI aimed much too high - that is the most common problem with
>government contracts. They rarely take the approach "start low, with what's
>available, and improve gradually", they always want the big thing, and are
>willing to wait ten years and hundred millions, and then scrap the whole
>thing.
>
>The harder thing is to prevent abuse of such a system. The easy part is to
>expire and clear files, though probably nothing will ever get deleted (at
>least searching should require a sort of "way back machine", i.e. not in
>the normal context). However, criminal officers can use such a virtual file
>and the search functions to hide. Therefore, each file should also keep
>track of who viewed it - if an officer not in charge views a particular
>file quite often, there's a suspect. Version control is also obvious. That
>all this should happen within a VPN, too.
>
>The web is a huge public general purpose information system, making a
>specialized in-house variant should really not take that much. The search
>engine companies sell their technology, they'll probably be happy with
>ideas how to make things better (a real-world Google could benefit from a
>thesaurus, too).

You're concerned about civil liberties, which is understandable.
That's a much bigger problem than the FBI's proposed
system--computerized collections of personal data or allegations about
indivduals (which the internet already is) and the ability to search
them (which google already is) are inimical to privacy.

The problem for the customer of record here is not civil liberties,
but security. The very same mechanisms that have made the internet so
stunningly powerful (flat resource space, amazing indexes, universal
access) are unacceptable for secure systems.

The security regimes with which I have been acquainted lean heavily on
strict compartmentalization by need to know. That means that access
paths all have to be laid out rigidly and with significant
foreknowledge about what metadata need to be visible--and it means the
metadata have to be tediously created by hand. I don't know how the
FBI has proposed to deal with compartmentalization, but you can't just
dump everything out there in a flat resource space, index it, and let
people do their own searches.

If you didn't have to worry about security, your proposal would be
magic. Digitize everything (although agents have apparently been
resistant to the step of putting things into a scanner), put it where
it can be indexed, index it, and let agents learn to think the way
that people who use and contribute to open source do.

The FBI didn't think this part through clearly, either, and figured
that, if you can correlate the entire planet with a quarter million
PC's (as in google), you should probably be able to cope with just the
FBI's case files for $170 Million and no special magic?

RM

From:	Bernd Paysan
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Sat, 15 Jan 2005 15:46:48 +0100

Robert Myers wrote:
> The security regimes with which I have been acquainted lean heavily on
> strict compartmentalization by need to know. That means that access
> paths all have to be laid out rigidly and with significant
> foreknowledge about what metadata need to be visible--and it means the
> metadata have to be tediously created by hand. I don't know how the
> FBI has proposed to deal with compartmentalization, but you can't just
> dump everything out there in a flat resource space, index it, and let
> people do their own searches.

As far as I know the FBI (and even more the secret services) have troubles
with "left hand doesn't know what right hand does". This is a consequence
of security regimes like the "need to know" principle. Since the FBI is a
federal police, the "need to know" regime should happen at the border to
the FBI, not within. Crimes that are no concern to the FBI don't go into
the FBI database, and crimes that do should allow to be linked relatively
freely. Organized crime needs correlation between seemingly unrelated
cases. Organized crimes also requires surveillance of the officers in
charge, as the combination of "need to know" and corruption is an easy way
for organized crime to continue their work even though the FBI has been set
on them.

The "need to know" principle does not help civil liberties. The Gestapo was
clearly organized that way, too. It's a two-sided sword - it helps to hide
crimes by those organizations as well as it helps to hide well enough
organized criminals.

Systems like that need boundaries. Some material needs to be classified
internally, too. However, it's dangerous to allow material to be locked in
(like the notebook from the suspected 9/11 highjacker which was caught a
month before). I'm quite sure that the inner boundaries, the virtual
cubicle walls inside the FBI are there for the same reasons many
programmers in classical corporate environments don't want other people to
look into their sources: They are there as job protection mean, to hide
errors, not to help nation security or civil liberties.

Sure, the database should not be searched that easily. At least a "four
eyes" approach or something like that is needed. Internal communication
within the team may be concealed from non-members (the same thing is even
possible on Sourceforge). For things like that (mailing lists), you don't
need special infrastructure, not even for a very tough security regime on
those mails (OpenPGP or S/MIME is clearly sufficient).

> If you didn't have to worry about security, your proposal would be
> magic. Digitize everything (although agents have apparently been
> resistant to the step of putting things into a scanner), put it where
> it can be indexed, index it, and let agents learn to think the way
> that people who use and contribute to open source do.

Hm, requires at least three silver bullets ;-). FBI agents are far away from
open source contributors.

--
Bernd Paysan
"If you want it done right, you have to do it yourself"
http://www.jwdt.com/~paysan/

From:	Casper H.S. Dik
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	15 Jan 2005 15:13:54 GMT

Bernd Paysan writes:

>Sure, the database should not be searched that easily. At least a "four
>eyes" approach or something like that is needed. Internal communication
>within the team may be concealed from non-members (the same thing is even
>possible on Sourceforge). For things like that (mailing lists), you don't
>need special infrastructure, not even for a very tough security regime on
>those mails (OpenPGP or S/MIME is clearly sufficient).

Data would need to be labeled so only sufficient cleared personel
and personel with a stated interest in certain data can see it;
that, of course, makes searching/fishing a task which is not something
everybody is allowed to do, at least not in the whole database.

Making sure that some part of the system knows who accessed what and
when is also very important: it serves both as a deterrent from abuse
and may allow tracing leaked documents.

>> If you didn't have to worry about security, your proposal would be
>> magic. Digitize everything (although agents have apparently been
>> resistant to the step of putting things into a scanner), put it where
>> it can be indexed, index it, and let agents learn to think the way
>> that people who use and contribute to open source do.

>Hm, requires at least three silver bullets ;-). FBI agents are far away from
>open source contributors.

A lot of the data the FBI gathers is not computerized; to go from
hand written notes to a searchable index is quite a step.

Casper

From:	del cecchi
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Sat, 15 Jan 2005 20:05:51 -0600

"Bernd Paysan" wrote in message
news:ovmnb2-rqa.ln1@vimes.paysan.nom...
> Robert Myers wrote:
> > The security regimes with which I have been acquainted lean heavily
on
> > strict compartmentalization by need to know. That means that access
> > paths all have to be laid out rigidly and with significant
> > foreknowledge about what metadata need to be visible--and it means
the
> > metadata have to be tediously created by hand. I don't know how the
> > FBI has proposed to deal with compartmentalization, but you can't
just
> > dump everything out there in a flat resource space, index it, and
let
> > people do their own searches.
>
> As far as I know the FBI (and even more the secret services) have
troubles
> with "left hand doesn't know what right hand does". This is a
consequence
> of security regimes like the "need to know" principle. Since the FBI
is a
> federal police, the "need to know" regime should happen at the border
to
> the FBI, not within. Crimes that are no concern to the FBI don't go
into
> the FBI database, and crimes that do should allow to be linked
relatively
> freely. Organized crime needs correlation between seemingly unrelated
> cases. Organized crimes also requires surveillance of the officers in
> charge, as the combination of "need to know" and corruption is an easy
way
> for organized crime to continue their work even though the FBI has
been set
> on them.

Sorry Charlie. You must not have gotten the memo about the wall between
criminal and national security work at the FBI. Sent by Garfolo(Not
sure about the spelling, worked for Reno)
>
> The "need to know" principle does not help civil liberties. The
Gestapo was
> clearly organized that way, too. It's a two-sided sword - it helps to
hide
> crimes by those organizations as well as it helps to hide well enough
> organized criminals.

You also don't know about the fruit of the poisoned tree and all that.
American Blind justice is almost too bizarre to think about.
>
> Systems like that need boundaries. Some material needs to be
classified
> internally, too. However, it's dangerous to allow material to be
locked in
> (like the notebook from the suspected 9/11 highjacker which was caught
a
> month before). I'm quite sure that the inner boundaries, the virtual
> cubicle walls inside the FBI are there for the same reasons many
> programmers in classical corporate environments don't want other
people to
> look into their sources: They are there as job protection mean, to
hide
> errors, not to help nation security or civil liberties.

Bzzt, thanks for playing. Read the accounts of the arrest and trial of
Z Massoui ( the purported 20th hijacker). His hard drive couldn't be
searched. Would violate his rights.

>
> Sure, the database should not be searched that easily. At least a
"four
> eyes" approach or something like that is needed. Internal
communication
> within the team may be concealed from non-members (the same thing is
even
> possible on Sourceforge). For things like that (mailing lists), you
don't
> need special infrastructure, not even for a very tough security regime
on
> those mails (OpenPGP or S/MIME is clearly sufficient).
>
I don't think you should talk about stuff you don't know about. They
already have a database. Hell, they probably have 17 databases. All
living on servers somewhere that are updated overnight.

snip

From:	Robert Myers
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Fri, 14 Jan 2005 13:16:28 -0500

Del Cecchi wrote:

>>
> How about a nice Blue Gene based system like the Mayo Clinic is building
> for genomic medicine and mining patient records? :-)
>

Blue Gene gets more amazing all the time. It now does virtualization? ;-).

A z-series mainframe, I would have thought, with virtualization on
virtualization to monitor and to trap suspicious behavior aimed at such
a high-value target, but what do I know?

If vmware can virtualize garden-variety x86, I'm sure that IBM can
virtualize the Power-based Blue Gene. What's one more ambitious goal,
more or less?

RM

From:	Bernd Paysan
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Sat, 15 Jan 2005 17:51:51 +0100

Del Cecchi wrote:
> Maybe they shouldn't have hired the low bidder. And the government gets
> what they ask for in the contract. Maybe they asked for the wrong
> thing, or maybe they asked for the undoable.

According to Heise online, one of the problems the FBI has is that the first
deliverable doesn't work perfect. Duh. This is to be expected, software
simply doesn't do that. An iteration cycle is necessary. Government
contracts are especially difficult to deal with, since the customer both
doesn't know what he wants *and* wants a perfect solution on the first
shot. Commercial development is easier, since while the customer there
doesn't know what he wants either, a perfect solution is not necessary.
Economic pressure makes sure that the customer will live with the 90%
solution, and allows for gradual improvement.

According to recent news on Heise, FBI also doesn't use "Carnivore", but
some unnamed commercial systems. They don't seem to have much luck with the
software development they initiate.

> Really wants one want to have them take over health care, eh?

Hm, the US system, as it is now, is nothing to be proud of, either. And you
can't say that the state is not already involved deeply - the FDA does such
weird things on pharmacy prices and availability that even MCs go to Canada
to buy cheaper drugs (or drugs unavailable in the USA).

So: given how much the state now already manages to mess up health care, no,
you wouldn't want more of that.

--
Bernd Paysan
"If you want it done right, you have to do it yourself"
http://www.jwdt.com/~paysan/

From:	jgrove24 at hotmail.com
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	15 Jan 2005 13:59:47 -0800

jgrove24@hotmail.com wrote:
> Robert Myers wrote:
> > Greetings,
> >
> > News articles say FBI is probably going to scrap its $170 Million
> > Virtual Case File system and start over from scratch.
> >
> > The problem is characterized as a software issue. Not being an
> > enterprise architect, I would have thought the problem was an
> enterprise
> > architecture issue: hardware, software, and mostly organizational
> wetware.
> >
> > I'm posting here because of the number of posters and lurkers with
> > significant architectural experience in mainframe and enterprise
> systems.
> >
>
> The features the FBI looked for are easy using UNIX, and a tool like
> Documentum. They were looking to manage CASE FILES, you know like
paper
> documents. And the ability to search or GREP in UNIX. A Grand Jury
> examination of this folly is in order.
>
> JG

Actually, I'm read usenet in GOOGLE, what the FBI wanted is basically
GOOGLE features.

JG

From:	Robert Myers
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Sat, 15 Jan 2005 17:40:08 -0500

On 15 Jan 2005 13:59:47 -0800, jgrove24@hotmail.com wrote:

>
>Actually, I'm read usenet in GOOGLE, what the FBI wanted is basically
>GOOGLE features.
>

The problem is that the part of the problem you can see--the part the
looks like the internet and google--is sufficiently non-trivial that
it is successfully masquerading as the whole problem to you.

The FBI wants data relevant to terrorism to be seen instantaneously
(as opposed to the currently claimed 24-hour cycle, which at that I
don't believe)--google can't do that.

There's lots of stuff on the web, the so-called deep web, that you
can't see with google, and the Virtual Case File would have to look
like a deep web so that it didn't pose unacceptable security risks.

Google can't manage document access or audit changes and versions.
The tools open source contributors use can do many of those things
nicely, but if FBI agents could function at that level of competence,
they could probably build the system themselves on the fly.

I wonder if something very similar to the false intuition you have
about this problem doesn't explain the whole wasted $170 million. It
looks easy, so it must be easy. Put it out on an RFP, take a
plausible bid, and wait for your own ready-made personal internet and
google to appear. Piece of cake.

RM

From:	mike
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Mon, 17 Jan 2005 05:30:37 GMT

There are many organizational and cultural hurdles for governmental
organizations to overcome when developing large systems. However this
posting implies one problem that does not exist.

With the Internet there is no way for GOOGLE to know immediately when a web
page is created or updated. Therefore, GOOGLE must scan the whole net, or
as much as it can access, every few days or weeks to keep its indexes fairly
current. Google can assign a high priority to some sites based on user
access frequency and site volatility. These sites can be scanned every day,
hour or even every few minutes. On the other hand changes to less important
sites will not be reflected in the index for weeks or even months. There is
no possibility for a "real time" index.

In a closed government system like the FBI case database the software that
enters or modifies a case file can send a message directly to the indexing
engine. Further unlike GOOGLE the indexing engine does not have to waste
cycles re-indexing old entries that have not been changed. Therefore a near
real time index, within a few hundred milliseconds, can be maintained.

Mike Sicilian

"Robert Myers" wrote in message
news:c76ju01ic19dv5h8efai6c021cqvol77f1@4ax.com...
> On 15 Jan 2005 13:59:47 -0800, jgrove24@hotmail.com wrote:
>
>>
>>Actually, I'm read usenet in GOOGLE, what the FBI wanted is basically
>>GOOGLE features.
>>
>
> The problem is that the part of the problem you can see--the part the
> looks like the internet and google--is sufficiently non-trivial that
> it is successfully masquerading as the whole problem to you.
>
> The FBI wants data relevant to terrorism to be seen instantaneously
> (as opposed to the currently claimed 24-hour cycle, which at that I
> don't believe)--google can't do that.
>
> There's lots of stuff on the web, the so-called deep web, that you
> can't see with google, and the Virtual Case File would have to look
> like a deep web so that it didn't pose unacceptable security risks.
>
> Google can't manage document access or audit changes and versions.
> The tools open source contributors use can do many of those things
> nicely, but if FBI agents could function at that level of competence,
> they could probably build the system themselves on the fly.
>
> I wonder if something very similar to the false intuition you have
> about this problem doesn't explain the whole wasted $170 million. It
> looks easy, so it must be easy. Put it out on an RFP, take a
> plausible bid, and wait for your own ready-made personal internet and
> google to appear. Piece of cake.
>
> RM

From:	Stephen Fuld
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Mon, 17 Jan 2005 06:00:16 GMT

"mike" wrote in message
news:14IGd.10035$Vj3.7515@newssvr17.news.prodigy.com...
> There are many organizational and cultural hurdles for governmental
> organizations to overcome when developing large systems. However this
> posting implies one problem that does not exist.
>
> With the Internet there is no way for GOOGLE to know immediately when a
> web page is created or updated. Therefore, GOOGLE must scan the whole
> net, or as much as it can access, every few days or weeks to keep its
> indexes fairly current. Google can assign a high priority to some sites
> based on user access frequency and site volatility. These sites can be
> scanned every day, hour or even every few minutes. On the other hand
> changes to less important sites will not be reflected in the index for
> weeks or even months. There is no possibility for a "real time" index.
>
> In a closed government system like the FBI case database the software that
> enters or modifies a case file can send a message directly to the indexing
> engine. Further unlike GOOGLE the indexing engine does not have to waste
> cycles re-indexing old entries that have not been changed. Therefore a
> near real time index, within a few hundred milliseconds, can be
> maintained.

Precisely. And the other technical problems seem fairly easily solvable.
For instance, I don't doubt that by sending some money to Google for a
special system for them selves, Google could easily take each query,
optionally look up all the words in a thesaurus and modify the queries to
include either the original word or the synonym. As for getting all of the
existing paper information into the database, Google has just commenced
scanning a huge amount of data, from several libraries into searchable form,
so this is right up their alley.

Some people mentioned the security issues. ISTM that either you allow FBI
agents access to the data or you don't. If you do, then there is no
problem. If you don't, then you can't say that you have "broken the stove
pipes" or "broken down the walls", as they claim to be doing. If there is
some particular need, again, having some code on each document and making
the search or display software aware of the codes seems like it could be
made to work.

While I agree that trying to use the existing Google system exactly as it is
now wouldn't work, but ISTM that it is so close, and the modifications
should cost a tiny fraction of the $157 million, that it is worth trying.

--
- Stephen Fuld
e-mail address disguised to prevent spam

From:	mike
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Mon, 17 Jan 2005 06:35:42 GMT

Hold on a second!!

I just said that real time index maintenance was not the issue, I did not
mean to imply that GOOGLE and a few million dollars would be a full function
case system for the FBI. I have not even seen a clear statement of system
requirements.

However, just thinking out loud, you can not just let every FBI agent look
at every file with no controls and no log of access. If Al Queda manages to
suborn just one agent they could find out exactly what the FBI was looking
at and the names of all informants and puff! the whole FBI is useless.

I would expect the FBI needs a system that will look at each case and notify
the agents on that case of other cases and documents that might be relevant.

Further, supervisors should get detailed reports on system usage so they can
verify the appropriateness of data requests as relevant to the assignments
of their staff.

Also, the system should have a "work flow management" component like call
centers and large help desk operations use, so that new data entry
transactions needing investigation or analysis are automatically assigned
and progress tracked. The FBI is a big organization and without such a
facility, too much will fall between the cracks.

One of the biggest problems with government system procurement is that many
of these features can be found in commercial packages. When available, they
should be used as subsystems rather than reinvented.

Mike Sicilian

"Stephen Fuld" wrote in message
news:QvIGd.33750$w62.13950@bgtnsc05-news.ops.worldnet.att.net...
>
> "mike" wrote in message
> news:14IGd.10035$Vj3.7515@newssvr17.news.prodigy.com...
>> There are many organizational and cultural hurdles for governmental
>> organizations to overcome when developing large systems. However this
>> posting implies one problem that does not exist.
>>
>> With the Internet there is no way for GOOGLE to know immediately when a
>> web page is created or updated. Therefore, GOOGLE must scan the whole
>> net, or as much as it can access, every few days or weeks to keep its
>> indexes fairly current. Google can assign a high priority to some sites
>> based on user access frequency and site volatility. These sites can be
>> scanned every day, hour or even every few minutes. On the other hand
>> changes to less important sites will not be reflected in the index for
>> weeks or even months. There is no possibility for a "real time" index.
>>
>> In a closed government system like the FBI case database the software
>> that enters or modifies a case file can send a message directly to the
>> indexing engine. Further unlike GOOGLE the indexing engine does not have
>> to waste cycles re-indexing old entries that have not been changed.
>> Therefore a near real time index, within a few hundred milliseconds, can
>> be maintained.
>
> Precisely. And the other technical problems seem fairly easily solvable.
> For instance, I don't doubt that by sending some money to Google for a
> special system for them selves, Google could easily take each query,
> optionally look up all the words in a thesaurus and modify the queries to
> include either the original word or the synonym. As for getting all of
> the existing paper information into the database, Google has just
> commenced scanning a huge amount of data, from several libraries into
> searchable form, so this is right up their alley.
>
> Some people mentioned the security issues. ISTM that either you allow FBI
> agents access to the data or you don't. If you do, then there is no
> problem. If you don't, then you can't say that you have "broken the stove
> pipes" or "broken down the walls", as they claim to be doing. If there is
> some particular need, again, having some code on each document and making
> the search or display software aware of the codes seems like it could be
> made to work.
>
> While I agree that trying to use the existing Google system exactly as it
> is now wouldn't work, but ISTM that it is so close, and the modifications
> should cost a tiny fraction of the $157 million, that it is worth trying.
>
> --
> - Stephen Fuld
> e-mail address disguised to prevent spam
>

From:	Martin Rodgers
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Mon, 17 Jan 2005 08:08:24 +0000

mike wrote:

> Further, supervisors should get detailed reports on system usage so they can
> verify the appropriateness of data requests as relevant to the assignments
> of their staff.

That sounds like profiling of FBI agents. Oh, the irony!
--
http://www.wildcard.demon.co.uk You can never browse enough
Will write code that writes code that writes code for food

From:	Stephen Fuld
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Mon, 17 Jan 2005 08:09:07 GMT

"mike" wrote in message
news:21JGd.18525$by5.14061@newssvr19.news.prodigy.com...
> Hold on a second!!
>
> I just said that real time index maintenance was not the issue, I did not
> mean to imply that GOOGLE and a few million dollars would be a full
> function case system for the FBI.

OK.

> I have not even seen a clear statement of system requirements.

Nor have I, but we can do some intelligent speculation.

> However, just thinking out loud, you can not just let every FBI agent
> look at every file with no controls and no log of access.

I mentioned in my post the idea of codes to limit access. That doesn't seem
like a major issue. And today, Google can log each query and does for
statistical purposes.

> If Al Queda manages to suborn just one agent they could find out exactly
> what the FBI was looking at and the names of all informants and puff! the
> whole FBI is useless.

What are the controls today? I remember the one agent complaining that she
couldn't see if other cities had people with arabic names taking flight
lessons and this was not due to some security issues, but because the system
simply couldn't support it. Clearly, some people need access to all the
data and if they are compromised, then we have a Robert Hanson type
situation.

> I would expect the FBI needs a system that will look at each case and
> notify the agents on that case of other cases and documents that might be
> relevant.

I'm not sure how you could code the test for "relevant". But let's assume
you could do that. Then when a new case is created, or when there are
substantial updates, a "relevant information" query could be run. Then the
software could automatically send an e-mail to the agents on the other
cases.

> Further, supervisors should get detailed reports on system usage so they
> can verify the appropriateness of data requests as relevant to the
> assignments of their staff.

Sure. A simple extension of the logging function seems to meet that
requirement.

> Also, the system should have a "work flow management" component like call
> centers and large help desk operations use, so that new data entry
> transactions needing investigation or analysis are automatically assigned
> and progress tracked. The FBI is a big organization and without such a
> facility, too much will fall between the cracks.
>
> One of the biggest problems with government system procurement is that
> many of these features can be found in commercial packages. When
> available, they should be used as subsystems rather than reinvented.

Yes. I absolutely agree. And it is even the case that it may be faster to
slightly modify an existing system than to code from scratch.

--
- Stephen Fuld
e-mail address disguised to prevent spam

From:	Bernd Paysan
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Mon, 17 Jan 2005 12:54:50 +0100

Stephen Fuld wrote:
> Precisely. And the other technical problems seem fairly easily solvable.
> For instance, I don't doubt that by sending some money to Google for a
> special system for them selves, Google could easily take each query,
> optionally look up all the words in a thesaurus and modify the queries to
> include either the original word or the synonym.

Not just the query, the index, too. Certainly, the thesaurus must map equal
words when querying, but when they go in as equals in the map (index), the
map is denser and smaller.

> While I agree that trying to use the existing Google system exactly as it
> is now wouldn't work, but ISTM that it is so close, and the modifications
> should cost a tiny fraction of the $157 million, that it is worth trying.

ISTM, too.

--
Bernd Paysan
"If you want it done right, you have to do it yourself"
http://www.jwdt.com/~paysan/

From:	Stephen Fuld
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Mon, 17 Jan 2005 18:09:11 GMT

"Bernd Paysan" wrote in message
news:cllsb2-0nl.ln1@miriam.mikron.de...
> Stephen Fuld wrote:
>> Precisely. And the other technical problems seem fairly easily solvable.
>> For instance, I don't doubt that by sending some money to Google for a
>> special system for them selves, Google could easily take each query,
>> optionally look up all the words in a thesaurus and modify the queries to
>> include either the original word or the synonym.
>
> Not just the query, the index, too. Certainly, the thesaurus must map
> equal
> words when querying, but when they go in as equals in the map (index), the
> map is denser and smaller.

That is a good point that I missed. But I am not sure whether it is a good
idea. Is it sometimes necessary to ask for the particular word and
explicitly not one of its synonyms? If so, then you could specify it in the
query, but that wouldn't work if the index had already deleted the
information. I don't know if there is such a requriement though.

--
- Stephen Fuld
e-mail address disguised to prevent spam

From:	Andrew Reilly
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Mon, 17 Jan 2005 22:45:28 +1100

On Mon, 17 Jan 2005 06:30:37 +0000, mike wrote:

> In a closed government system like the FBI case database the software that
> enters or modifies a case file can send a message directly to the indexing
> engine.

kqueue, famd, snapshots ?

No, I know that's probably not the whole answer...

--
Andrew

From:	Robert Myers
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Mon, 17 Jan 2005 08:59:26 -0500

On Mon, 17 Jan 2005 05:30:37 GMT, "mike" wrote:

>
>In a closed government system like the FBI case database the software that
>enters or modifies a case file can send a message directly to the indexing
>engine. Further unlike GOOGLE the indexing engine does not have to waste
>cycles re-indexing old entries that have not been changed. Therefore a near
>real time index, within a few hundred milliseconds, can be maintained.
>

Well, sure. People know how to keep idices updated. In order to
index something, the data have to be visible to the indexing system in
a secure, consistent way that the indexing system understands. Many a
slip 'twixt the cup and the lip. That's why I don't believe the
current FBI system is only ever 24-hours out of date. When they find
another roomful of boxes, it'll have been however many months or years
out of date that roomful of boxes was hiding.

This from my inbox

--------------------------Advertisement-------------------------------
: Watch
everything work together and see change coming a mile away.
Reduce the number of IT elements in your network. Eliminate
customisation. Automate change. Use standard technologies and
interfaces. Create reusable components. Implement consistent
processes.
Interact with any system, anywhere, always. Build architectures
modularly. Virtualise your systems. Change one element without
impacting the entire network. Build a dynamic link between business
and IT. Connect applications and processes inside and out.

*******

Above all, minimize difficulties, oversell, and watch people go back
to pad and pencil, or at least bypass your nifty enterprise
architecture and keep things squirreled away in private stashes, maybe
even on a computer--a PC or a laptop. And, oh yes, watch as revenue
and jobs disappear from the industry.

Maybe there is a National Academies report that would answer the
question in my OP.

RM

From:	krasicki
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	18 Jan 2005 19:46:04 -0800

Stephen Sprunk wrote:
> "krasicki" wrote in message
> news:1106021732.876074.76930@c13g2000cwb.googlegroups.com...
> > The intent [just guessing] is to anticipate what might happen, who
> > might be inviolved, what might prevent the event rather than the
more
> > pedestrian anthropology of what happened, who did it, give me
instant
> > suspects. With all due respect to the latter activity, and
considering
> > the consequences, the former problem will not be solved COTS style.
>
> If that was the intent, then why the heck is it called "Virtual Case
File"?
> The name strongly implies they want an electronic document storage
and
> management system.

There are often massive differences between what something sounds like
and what something is.

As a taxpayer and wearing the hat of an information analyst, I would be
furious if $150+M were being spent to manage legacy FBI paperwork
because that would mean that 'terror' in the eyes of the FBI
constituted the imminent need to file and retrieve paperwork.

Aside from the sheer absurdity that pre-911 case file information
schemas are sufficient to capture the kind of data necessary for
analyzing terrorist threats [instead of, say, political tom-foolery]
there is probably no canonic and extendable information schema that
maps historic case file differences, cross-agency differences, and
intra-agency diffences of even existing data. Up-front questions about
such a project is what kind of documents do you have, what are they
worth, how clean is the quality of the stuff, what would you be
interested in, and so on.

And the reason you would ask these questions is to introduce rigor to
the information schema - when an agent needs a bit of information that
bit of information has to have a well-known, certain meaning.

But even this is presumptive that the agency has interest in mining the
nuances of existing data.

The specification could just as well mean that the paper inventory of
index cards needs to be physically retrievable. Modern libraries are
moving away from Dewey Decimal systems to inventory control systems in
which physical payload is stored and retrieved from an automated
container management system. Material is put into a container, the
contents identified, and the system carries it away until the material
is requested. No more messy file cabinets or bookshelves to browse.

>
> Another project might use VCF as a source of information to decide
who
> "might" be a terrorist, but that doesn't seem like the purpose of VCF
> itself.
>

Sounds like the FBI still has no imagination if that bullet point
didn't make the 'wish list';-)

From:	krasicki
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	16 Jan 2005 20:11:10 -0800

Very good points.

Right the first part of the problem is a canonic case file, the second
part is making it virtual, the third part - realtime, the fourth -
realtime rollback when it's wrong, and so on.

Architecturally, we are talking about the largest in-memory database
ever attempted and we're talking about synchronization features that
haven't been dreamed of with the initial investment. As you say, this
is almost certainly vendor driven, government payola driving the train.

One of the most severe problems that we as a nation face in truly
beginning to prepare ourselves against the war on terror (let's face it
- the Bush administration are klowns) is that the RFPs for far too many
systems have overt, vendor driven agendas built into the spec. Often
the vendor insists only their inside consultants can do the work. The
contract award this becomes self-insulating for the vendor or family of
vendors and no fresh air ever gets in to disinfect the project from
poor management, honest review, and so on.
And you are right in observing that Google is not the answer.

From:	jgrove24 at hotmail.com
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	17 Jan 2005 13:47:15 -0800

>
> Finally, if they can't understand the requirements of what they want
-
> then managing [people AND technology] will bring any such system to
its
> knees during peak vacation times. $170M is a drop in the bucket for
a
> meta-enterprise system requirement such as this.

Assuming a billing rate of $250K/year, what exactly was produced for
680
person-years of work?

per LA Times:

A preliminary report from Aerospace Corp., a federally funded nonprofit
research firm in El Segundo hired by the FBI to assess its options, has
identified commercially available programs that could meet the FBI's
requirements, sources familiar with the study said. Using such programs
would also enable the FBI to integrate its software with that of other
agencies doing similar work - a far more complicated task if it chose
to stick with a custom product.
......
In 1995, the agency's vast paper files were partly loaded into a
searchable electronic database. That system was soon found to be
inefficient and riddled with software bugs, and some top officials
never adopted it.

Until recently, much of the rest of the agency - including some field
agents - lacked such basic features as e-mail.

After the Sept. 11 attacks, the FBI's hardware, software and
communications networks, built up at the cost of hundreds of millions
of dollars, were revealed as severely outmoded.
.....
The linchpin of the planned computer system was the Virtual Case File
software, designed for use on fast computers and networks to make it
widely accessible across the FBI's scores of offices. The plan was to
include images, video and sounds, crucial components of a modern
investigation.
....and CNN
The current program requires FBI personnel to manually enter, print,
sign and scan their information into the "investigative data
warehouse."
The new software program was supposed to allow agents to pass along
along intelligence and criminal information in real time.

From:	Robert Myers
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Mon, 17 Jan 2005 17:30:47 -0500

On 17 Jan 2005 13:47:15 -0800, jgrove24@hotmail.com wrote:

>
>>
>> Finally, if they can't understand the requirements of what they want
>-
>> then managing [people AND technology] will bring any such system to
>its
>> knees during peak vacation times. $170M is a drop in the bucket for
>a
>> meta-enterprise system requirement such as this.
>
>Assuming a billing rate of $250K/year, what exactly was produced for
>680
>person-years of work?
>
>per LA Times:
>
>A preliminary report from Aerospace Corp., a federally funded nonprofit
>research firm in El Segundo hired by the FBI to assess its options, has
>identified commercially available programs that could meet the FBI's
>requirements, sources familiar with the study said. Using such programs
>would also enable the FBI to integrate its software with that of other
>agencies doing similar work - a far more complicated task if it chose
>to stick with a custom product.

Aerospace Corp is at least a disinterested party. Whether it has the
expertise to do this particular job is another question. It certainly
doesn't do enterprise IT or (AFAIK) supervise enterprise IT-type
contracts.

HP blamed a $208 million quarterly loss for its enterprise storage and
server division on a new inventory and logsitics control system last
year. No security and no criminal case file issues. What would it
have cost HP to do it right the first time? Anything less than $208
million would have been a bargain. I wouldn't have wanted to be
selling HP enterprise services when that was going down.

Rather than what do you need all this money for (which is surely the
question being asked), a better place to start would be how expensive
is it to get it wrong. The Federal government has gotten so many of
these systems wrong, like the social security computer system, that
it's tempting to say derisive things about the US government IT. With
the other examples in this thread, maybe we should just say derisive
things about government IT. But then HP can't get its own system
right. One could say derisive things about HP? Already done.

Or maybe the problem is just hard and people are chronically
underestimating the actual difficulty and cost of doing it right. The
"Oh, we'll just do this and this and that, and it'll be done"
handwaving has obviously been done to an expensive fare-thee-well.

RM

From:	Stephen Fuld
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Tue, 18 Jan 2005 01:54:55 GMT

"Robert Myers" wrote in message
news:7jdou09inojg2a80mufa7a1sbkpr71dohn@4ax.com...

snip

> Rather than what do you need all this money for (which is surely the
> question being asked), a better place to start would be how expensive
> is it to get it wrong. The Federal government has gotten so many of
> these systems wrong, like the social security computer system, that
> it's tempting to say derisive things about the US government IT.

What is wrong with the Social Security computer system(s)? There were lots
well documented screwups in implementing the SSI system in the 1970s (and I
can talk about that), but I haven't heard much since. The Social Security
system is usually cited as the epitome of government efficiency, costing
taxpayers only 1-2 percent of benefit costs in administrative expense. Full
Disclosure - I used to work for the Social Security Administration in the
1970s.

--
- Stephen Fuld
e-mail address disguised to prevent spam

From:	Robert Myers
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Mon, 17 Jan 2005 21:44:48 -0500

On Tue, 18 Jan 2005 01:54:55 GMT, "Stephen Fuld"
wrote:

>
>"Robert Myers" wrote in message
>news:7jdou09inojg2a80mufa7a1sbkpr71dohn@4ax.com...
>
>snip
>
>> Rather than what do you need all this money for (which is surely the
>> question being asked), a better place to start would be how expensive
>> is it to get it wrong. The Federal government has gotten so many of
>> these systems wrong, like the social security computer system, that
>> it's tempting to say derisive things about the US government IT.
>
>What is wrong with the Social Security computer system(s)? There were lots
>well documented screwups in implementing the SSI system in the 1970s (and I
>can talk about that), but I haven't heard much since. The Social Security
>system is usually cited as the epitome of government efficiency, costing
>taxpayers only 1-2 percent of benefit costs in administrative expense. Full
>Disclosure - I used to work for the Social Security Administration in the
>1970s.

August 2003 google groups comp.arch "social security." I may well
have different federal fiascos conflated by now. More than one
problem system was discussed.

RM

From:	Stephen Fuld
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Tue, 18 Jan 2005 06:32:16 GMT

"Robert Myers" wrote in message
news:ettou0lqsk6p15kpic17t3jfi349do2enh@4ax.com...
> On Tue, 18 Jan 2005 01:54:55 GMT, "Stephen Fuld"
> wrote:
>
>>
>>"Robert Myers" wrote in message
>>news:7jdou09inojg2a80mufa7a1sbkpr71dohn@4ax.com...
>>
>>snip
>>
>>> Rather than what do you need all this money for (which is surely the
>>> question being asked), a better place to start would be how expensive
>>> is it to get it wrong. The Federal government has gotten so many of
>>> these systems wrong, like the social security computer system, that
>>> it's tempting to say derisive things about the US government IT.
>>
>>What is wrong with the Social Security computer system(s)? There were
>>lots
>>well documented screwups in implementing the SSI system in the 1970s (and
>>I
>>can talk about that), but I haven't heard much since. The Social Security
>>system is usually cited as the epitome of government efficiency, costing
>>taxpayers only 1-2 percent of benefit costs in administrative expense.
>>Full
>>Disclosure - I used to work for the Social Security Administration in the
>>1970s.
>
> August 2003 google groups comp.arch "social security." I may well
> have different federal fiascos conflated by now. More than one
> problem system was discussed.

I checked and there was much confusion of several different projects and
several more situations. The closest to a "fiasco" was the one I described
back then, which was an attempt to redesign the entire Social Security data
processing systems in one fell swoop. It was an in house effort and they
had dozens of analysts working one it, but it never got to the coding stage
before the management came to their senses and realized that it wouldn't
work. So it was not a system that was implemented and "gotten wrong".
Other than that, I can't find anything that matches the criteria you had in
your post.

Not that there haven't been a substantial number of such "fiasco"s in the
federal government. :-(

--
- Stephen Fuld
e-mail address disguised to prevent spam

From:	Robert Myers
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Tue, 18 Jan 2005 10:06:02 -0500

On Tue, 18 Jan 2005 06:32:16 GMT, "Stephen Fuld"
wrote:

>
>I checked and there was much confusion of several different projects and
>several more situations. The closest to a "fiasco" was the one I described
>back then, which was an attempt to redesign the entire Social Security data
>processing systems in one fell swoop. It was an in house effort and they
>had dozens of analysts working one it, but it never got to the coding stage
>before the management came to their senses and realized that it wouldn't
>work. So it was not a system that was implemented and "gotten wrong".
>Other than that, I can't find anything that matches the criteria you had in
>your post.
>
>Not that there haven't been a substantial number of such "fiasco"s in the
>federal government. :-(

Not everything is a failure, I'm sure. What I partly had in mind
about social security was its more recent move to decentralized data
processing. I know I had read some complaints, but now that I look
over what's quickly available, it seems to have worked--certainly not
a fiasco.

It's not the Federal Government, but Lockheed is bragging about how
much it has reduced paperwork in its contract submissions by putting
everything (including drawings in editable format) on computers. Come
to think of it, the Aerospace Corporation probably does have some
experience with a system with many of the capabilities the FBI wants,
including the capacity to handle sensitive documents. Wonder how well
it really works?

It's difficult for someone not actually involved even to guess what's
going wrong, how much should be forseeable as being at least very
risky, and how much is just the misery of dealing with complex
systems.

RM

From:	Some Guy Who Doesn't Want Spam
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Thu, 20 Jan 2005 16:41:24 -0600

jgrove24@hotmail.com wrote:
> ...and CNN
> The current program requires FBI personnel to manually enter, print,
> sign and scan their information into the "investigative data
> warehouse."
> The new software program was supposed to allow agents to pass along
> along intelligence and criminal information in real time.

It's worse than you think... If you poke around a bit, you discover that
the "Investigative Data Warehouse" was a proof-of-concept piece intended
mostly for A: evaluating search tools, and B: figuring out what analysts
needed. It went online in summer of '04, it worked beautifully,
and the analyst test group using it told other analysts, who promptly
glommed on to the first (and at this point, only) working
multiple-source search capability.

Later in the CNN article, it mentions that there are 6000 operational users
of that system... which is pretty impressive for a system designed for a test
group of about 200. (This may also be why it's so awkward to enter data into
the system... It may not have been intended for user updates of data.)

From:	krasicki
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	17 Jan 2005 20:15:32 -0800

Stefan,

It is precisely what I wanted to say and I agree with you that the
current political opportunists will never get it right although some
of the problem is endemic to all governments.

In the United States too little thought [in fact none] goes into
ensuring a harmony between technology and governance [rights and law
inclusive].

What the FBI is asking for is technically feasible, it will never be
cheap until the extreme expense of doing it the first time is spent,
and it will not be like Google, grep, or many of the familiar utilities
being argued here.

It will require non-trivial artificial intelligence components,
robotics, and human exo-skeleton sensory extensions to gather and
process information. Although, Google search queries like, "Find me a
terrorist I can arrest." are quaint wish list items what is really
needed is a push mechanism that triangulates a multiplicity of weighed
behavioral, inferred, and heuristically enhanced, pattern-matching
analysis tools. You can't begin to talk about a Virtual Case File
until an inventory of these techniques is accounted for and
expectations level set.

This problem is not profoundly different from NASA's 'search for life'
techniques and certain gaming algorithms may be useful in unorthodox
ways.

The intent [just guessing] is to anticipate what might happen, who
might be inviolved, what might prevent the event rather than the more
pedestrian anthropology of what happened, who did it, give me instant
suspects. With all due respect to the latter activity, and considering
the consequences, the former problem will not be solved COTS style.

From:	Stephen Sprunk
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Tue, 18 Jan 2005 12:44:41 -0600

"krasicki" wrote in message
news:1106021732.876074.76930@c13g2000cwb.googlegroups.com...
> The intent [just guessing] is to anticipate what might happen, who
> might be inviolved, what might prevent the event rather than the more
> pedestrian anthropology of what happened, who did it, give me instant
> suspects. With all due respect to the latter activity, and considering
> the consequences, the former problem will not be solved COTS style.

If that was the intent, then why the heck is it called "Virtual Case File"?
The name strongly implies they want an electronic document storage and
management system.

Another project might use VCF as a source of information to decide who
"might" be a terrorist, but that doesn't seem like the purpose of VCF
itself.

S

--
Stephen Sprunk "Stupid people surround themselves with smart
CCIE #3723 people. Smart people surround themselves with
K5SSS smart people who disagree with them." --Aaron Sorkin

From:	Jan_Vorbrüggen
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Wed, 19 Jan 2005 09:23:12 +0100

> Another project might use VCF as a source of information to decide who
> "might" be a terrorist,

Solving that problem is only to a very small degree amenable to spending
more money. It would take years or, more likely, decades of research while
giving the researchers access to real-life data, for starters. And a lot of
others things in addition. I don't see that happening.

Jan

From:	Stephen Sprunk
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Thu, 20 Jan 2005 09:18:58 -0600

"Jan Vorbrüggen" wrote in message
news:356jnfF4j7q4kU1@individual.net...
> > Another project might use VCF as a source of information to decide who
> > "might" be a terrorist,
>
> Solving that problem is only to a very small degree amenable to spending
> more money. It would take years or, more likely, decades of research while
> giving the researchers access to real-life data, for starters. And a lot
of
> others things in addition. I don't see that happening.

I didn't mean to imply that such a task is actually possible with today's
knowledge, just that (based on the name) it doesn't sound like that was the
point of VCF.

S

--
Stephen Sprunk "Stupid people surround themselves with smart
CCIE #3723 people. Smart people surround themselves with
K5SSS smart people who disagree with them." --Aaron Sorkin

From:	Eugene Miya
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	18 Jan 2005 11:30:18 -0800

Just passing thru.

In article <1106073846.68565940416bbb820ea211806518463d@teranews>,
Stephen Sprunk wrote:
>"krasicki" wrote in message
>news:1106021732.876074.76930@c13g2000cwb.googlegroups.com...
>> The intent [just guessing] is to anticipate what might happen, who
>> might be inviolved, what might prevent the event rather than the more
>> pedestrian anthropology of what happened, who did it, give me instant
>> suspects. With all due respect to the latter activity, and considering
>> the consequences, the former problem will not be solved COTS style.
>
>If that was the intent, then why the heck is it called "Virtual Case File"?
>The name strongly implies they want an electronic document storage and
>management system.

This is largely in fact what it is and is meant to be.
I was asked and did finally visit the Bureau and briefly worked with
their Chief Scientist. The Bureau no longer has that position.
They did approach my agency for assistance but due to the distance
of my/our programmatic offices I recommended they talk to people at
Goddard.

It's called a case file going back to the Hoover days and one of the
things he did to revolutionize the Bureau (it's not called a bureaucracy
for nothing):
Hoover brought in 3 x 5 cards.

The Bureau is highly backlogged with FOIA requests. That should give
you a hint.

>Another project might use VCF as a source of information to decide who
>"might" be a terrorist, but that doesn't seem like the purpose of VCF
>itself.

Maybe later stages. That's a slightly different matter.

There is a talk abstract from a project we funded at UCB given by a
friend when she was at Berkeley but its no longer on line and one would
have to search the Internet Archive (unless it has been removed at DOJ
request).

In a way I feel sorry for the FBI, but in other ways I don't.
It's truly a case by case basis.

Whit Diffie (who is giving a Museum talk in 2 weeks) pegged Bureau
mentality in his book.

Gotta go, just passing thru.

Oh, is it possible? Hmmm. Well, likely you want to substitute other TLAs
and see if you think that.

--

From:	Robert Myers
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Tue, 18 Jan 2005 16:57:32 -0500

On 18 Jan 2005 11:30:18 -0800, eugene@cse.ucsc.edu (Eugene Miya)
wrote:

>
>Oh, is it possible? Hmmm. Well, likely you want to substitute other TLAs
>and see if you think that.

I like this article

http://www.floridatoday.com/news/space/stories/2004b/spacestoryN1010ROCKETSIDE.htm

Oct 10, 2004

Most defense space projects over budget

BY JOHN KELLY
FLORIDA TODAY

CAPE CANAVERAL -- The Delta 4 and Atlas 5 rockets aren't the only
space projects to cost more than the government expected.

Most Defense Department space projects are over budget, many by
billions of dollars.

In quarterly reports to Congress, the Pentagon reports at least $20
billion worth of overruns on its active space programs. That does not
count secret programs, such as spy satellites.

Then there's NASA. On average, the space agency's projects come in 45
percent over budget, the Congressional Budget Office recently
reported.

The simple explanation from the Air Force and others in the business:
"Space is hard."

The article goes on to speculation as to what's really happening that
might fit right into this thread (changing the subject from space to
computers, of course).

That fact that nothing seems to get done without going significantly
over budget isn't the same thing as saying that nothing ever gets
done, of course, but some of the ways things go wrong don't seem
likely to be peculiar to the FBI.

RM

From:	Eugene Miya
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	20 Jan 2005 10:52:11 -0800

In article <1v0ru0phjdv5571gmolugemaapjc7dt0hr@4ax.com>,
Robert Myers wrote:

I got email requested back for a moment.

>On 18 Jan 2005 11:30:18 -0800, eugene@cse.ucsc.edu (Eugene Miya)
>wrote:
>>Oh, is it possible? Hmmm. Well, likely you want to substitute other TLAs
>>and see if you think that.
>
>I like this article

While DOD is a TLA, it is more than well known that the DOD can throw
money almost wherever it wants. If you want to self select like this
proverbal guy under the street light looking for dropped keys at night,
you aren't going to think about things.

>http://www.floridatoday.com/news/space/stories...
>Most defense space projects over budget
This is a digression from your own question.

Try thinking about the usual (and less usual) conspiracy theory agencies
if you want to consider the realm of possibilities independent of the
Bureau's bureaucracy:

CIA Virtual Case File is even possible?
NSA Virtual Case File is even possible?
G[S]IA Virtual Case File is even possible?
NRO Virtual Case File is even possible?

And then you can consider "even possible?"
Certainly not hardware architecture.

--

From:	del cecchi
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Tue, 18 Jan 2005 18:55:27 -0600

"Robert Myers" wrote in message
news:1v0ru0phjdv5571gmolugemaapjc7dt0hr@4ax.com...
> On 18 Jan 2005 11:30:18 -0800, eugene@cse.ucsc.edu (Eugene Miya)
> wrote:
>
>
> >
> >Oh, is it possible? Hmmm. Well, likely you want to substitute other
TLAs
> >and see if you think that.
>
> I like this article
>
>
http://www.floridatoday.com/news/space/stories/2004b/spacestoryN1010ROCKETSIDE.htm
>
>
>
> Oct 10, 2004
>
> Most defense space projects over budget
>
> BY JOHN KELLY
> FLORIDA TODAY
>
> CAPE CANAVERAL -- The Delta 4 and Atlas 5 rockets aren't the only
> space projects to cost more than the government expected.
>
> Most Defense Department space projects are over budget, many by
> billions of dollars.
>
> In quarterly reports to Congress, the Pentagon reports at least $20
> billion worth of overruns on its active space programs. That does not
> count secret programs, such as spy satellites.
>
> Then there's NASA. On average, the space agency's projects come in 45
> percent over budget, the Congressional Budget Office recently
> reported.
>
> The simple explanation from the Air Force and others in the business:
> "Space is hard."
>
>
>
> The article goes on to speculation as to what's really happening that
> might fit right into this thread (changing the subject from space to
> computers, of course).
>
> That fact that nothing seems to get done without going significantly
> over budget isn't the same thing as saying that nothing ever gets
> done, of course, but some of the ways things go wrong don't seem
> likely to be peculiar to the FBI.
>
> RM

Duh. Where have these idiots been? How do you estimate the cost of
doing what has never been done? And is the pressure to estimate high or
low? It wasn't that the FBI program went over budget, but that they
went over budget and it still didn't even come close to working.

del

From:	Terje Mathisen
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Wed, 19 Jan 2005 07:46:21 +0100

del cecchi wrote:
> Duh. Where have these idiots been? How do you estimate the cost of
> doing what has never been done?

In the case of NASA, your best/only guess is to look at how you've
budgeted previous projects, vs how much they've really cost. This is the
only way I know to even guess at the required fudge factor. :-(

Of course, some things turn out to be effectively impossible, in which
case you need clear milestone targets, and the guts to kill projects
that aren't going to work out.

I'm running the 'IT Fire Department' in Hydro, which means that my group
get all the 'critical/totally new stuff/overrun/should have been working
last week' projects. We have an incredible success rate, partly because
I insist on splitting complicated development jobs into multiple clearly
delineated projects, with a stop/go decision after each part.

If it isn't going to work at all, I'd much rather find out sooner rather
than later.

(Yes, I do realize that this is hard to do when you have multiple
man-years of effort before you can have even a mock-up working. I stay
away from such projects. :-)

> And is the pressure to estimate high or low?

Here's the crux!

When all competing projects are going to submit a 'best case' guess,
instead of a maximum (90%?) likelyhood/median/error range number, your
own project pretty much have to do the same thing, otherwise you'll
never get of the ground (literally in NASA's case).

Another example from the other end of the spectrum:

I am leading the orienteering group at the corporate sports club, we
submit budgets each fall for the next year's activities. Three years in
a row we submitted exact budgets, i.e. we knew within about 10% what
both income and expense would be. All three years our budget got slashed
by 50%, because all other groups in the club had overbudgetted madly, so
everyone got slashed by the same percentage.

Last year we fixed this, not by doing a similar padding of the budget,
but by writing an enclosing letter stating how we knew this budget to be
exact (as well as _much_ lower than any other group's), pointing to the
very close correlation between the previous three year's original budget
and actual result, and that we would much rather keep on doing the best
budgetting job we knew how, than to become a part of the problem by
doing the same padding as everyone else.

It did work, we got almost exactly what we asked for. :-)

> It wasn't that the FBI program went over budget, but that they
> went over budget and it still didn't even come close to working.

Right.

Terje

--
-
"almost all programming can be viewed as an exercise in caching"

From:	Robert Myers
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Wed, 19 Jan 2005 08:48:04 -0500

On Tue, 18 Jan 2005 18:55:27 -0600, "del cecchi"
wrote:

>
>"Robert Myers" wrote in message
>news:1v0ru0phjdv5571gmolugemaapjc7dt0hr@4ax.com...
>> On 18 Jan 2005 11:30:18 -0800, eugene@cse.ucsc.edu (Eugene Miya)
>> wrote:
>>
>>
>> >
>> >Oh, is it possible? Hmmm. Well, likely you want to substitute other
>TLAs
>> >and see if you think that.
>>
>> I like this article
>>
>>
>http://www.floridatoday.com/news/space/stories/2004b/spacestoryN1010ROCKETSIDE.htm
>>

>>
>> The article goes on to speculation as to what's really happening that
>> might fit right into this thread (changing the subject from space to
>> computers, of course).
>>
>> That fact that nothing seems to get done without going significantly
>> over budget isn't the same thing as saying that nothing ever gets
>> done, of course, but some of the ways things go wrong don't seem
>> likely to be peculiar to the FBI.
>>
>
>Duh. Where have these idiots been? How do you estimate the cost of
>doing what has never been done? And is the pressure to estimate high or
>low? It wasn't that the FBI program went over budget, but that they
>went over budget and it still didn't even come close to working.
>

That (big money, no product, not even a working prototype) also
happened with the Sergeant York Gun, for example. In the case of the
Space Shuttle, it only ever worked in the sense that the goal posts
were moved very far from the way the program was sold (it really was
supposed to be a shuttle, like the New York-Boston shuttle, then a
pioneering service offered by Eastern Airlines). I really don't think
the problems are peculiar to the FBI.

As to the realities of goals and budgets for new projects, you
_surely_ must have at least a clue as to how that goes in large,
bureaucratic organization. The short time scale on which the budget
is completely redone for the US government means that promises have to
be sufficiently ambitious to survive the budget axe, and the budget
has to fit into whatever arbitrary constraints have been dictated.
That predicts chronic overpromising and underbudgeting, a prediction
that is borne out by experience.

One way to regularize the discussion of the FBI Case File system would
be to divide it into a part that is relatively routine and a part that
hasn't really been done before. The challenge for the relatively
routine part (a secure document management system) is probably getting
agents actually to use it without disrupting ongoing cases; i.e., it
really shouldn't be a "software problem." The non-routine part,
prognostic data mining, sounds like it's just burueacratic sales talk
at this point.

RM

From:	mike
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Wed, 19 Jan 2005 21:52:16 GMT

>The non-routine part,
> prognostic data mining, sounds like it's just burueacratic sales talk
> at this point.

Most of this thread keeps repeating the assumption that the FBI system was a
failure because it tried to do this and it probably is impossible with
current technology. We have not seen a real requirements specification and
this may or may not be in it. I wonder if we are over stating the goals.

Historically, investigative organizations have tried to find all the parts
of a crime and all individuals involved in a conspiracy by looking at every
contact and relationship. That is, they would interview all family members,
look at the phone bills and interview everyone that was called, look at the
credit card bills and examine travel patterns and talk to everyone who sold
something to the suspects, etc.

If I were trying to design a system for the FBI, I would concentrate on
automating data mining from this perspective rather than try to directly
predict criminal intent. If the system contained info on every suspected /
convicted criminal including terrorists and also contained detailed credit
card, phone, e-mail etc. transaction history you might be able to quickly
generate a "related suspect" list for further investigation and or
surveillance. The system would not say "hay these suspects are all taking
flying lessons, they intend to attack the World Trade Center". Rather, it
would say this suspected terrorist has been in contact with these 23 people
and 4 of them have received money transfers from a common account in Libya.

There are cultural and organizational challenges to successful system
development in any large organization and these are exaggerated in
government bureaucracies. However this more modest goal would seem to be
theoretically possible and an appropriate use of technology by the FBI.

Mike Sicilian

From:	Rick Jones
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Wed, 19 Jan 2005 18:21:29 GMT

del cecchi wrote:
> Duh. Where have these idiots been? How do you estimate the cost of
> doing what has never been done?

The same way we all estimate the performance of systems that don't
exist?-)

rick jones
--
Process shall set you free from the need for rational thought.
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to raj in cup.hp.com but NOT BOTH...

From:	Del Cecchi
Subject:	Re: [OT?] FBI Virtual Case File is even possible?
Date:	Wed, 19 Jan 2005 13:19:47 -0600

Rick Jones wrote:
> del cecchi wrote:
>
>>Duh. Where have these idiots been? How do you estimate the cost of
>>doing what has never been done?
>
>
> The same way we all estimate the performance of systems that don't
> exist?-)
>
> rick jones

Nope. I know what the non existant system is supposed to do and how it
is supposed to work, so I can simulate its performance. Actually
implementing or designing such a system maybe in fact be impossible or
require invention.

An example "Wouldn't it be nice if we had a system that never had a
cache miss. What would be the performance of such a system?"

Now, build one. :-)

From:	Terje Mathisen
Subject: