Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
Home
Discussion GroupsFormsForms ProgrammingQueriesModules / DAO / VBAReports / PrintingMacrosDatabase DesignSecurityConversionImporting / LinkingSQL Server / ADPMultiuser / NetworkingReplicationSetup / ConfigurationDeveloper ToolkitsActiveX ControlsNew UsersGeneral 1General 2
Access DirectoryToolsTutorialsUser Groups
Related Topics
SQL ServerOther DB ProductsMS OfficeMore Topics ...

MS Access Forum / General 2 / May 2008

Tip: Looking for answers? Try searching our database.

Upsizing?

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
David Portwood - 17 May 2008 16:00 GMT
I'm working a contract in the QA department of a well known multinational
car company. Their QA survey data is surprisingly disorganized and I
proposed an Access application to facilitate data collection and analysis.
When I began coding the app I ran into trouble storing the truly huge
volumes of data in Access tables.

Respected members of this group advised upsizing using another product as
backend. That might be a possibility but I have never administered a system
involving Access as a front end to something like a SQLServer (or maybe
Oracle) backend. I have other duties and I'm worred that I won't have time
to climb a learning curve. I see that Access includes an upsizing wizard,
which is encouraging.

But, learning curve aside, we are still dealing with hundreds of millions of
records. We will probably hit a billion before the end of the year. Even
using a better product as the back end, is it really possible to calculate
sums and averages across that much data? For instance, across a network,
would it not take hours to run a simple Totals query? I suppose it must be
doable or nobody would bother upsizing.

I understand that my question is extremely open ended and I certainly don't
want anyone writing a book in response to this post. It's actually unlikely
that I would be allowed to go down this road even if I could find the
personal courage to do so, but I would like to know just a little more so I
can at least discuss the matter intelligently while I'm apologizing to my
boss for making promises that I can't keep.

Thanks in advance.

David Portwood
Albert D. Kallal - 17 May 2008 18:04 GMT
> But, learning curve aside, we are still dealing with hundreds of millions
> of records. We will probably hit a billion before the end of the year.
> Even using a better product as the back end, is it really possible to
> calculate sums and averages across that much data?

The first question you have to ask is are you the only person out of 8
billion people on the planet earth that has this problem?  (answer = nope!).

> For instance, across a network, would it not take hours to run a simple
> Totals query? I suppose it must be doable or nobody would bother upsizing.

What is done in these cases is a term called in our industry that you might
have heard, and it called data warehousing. What happens is you take all
this data and put it into a server, and it munches and crunches totals over
given periods of time (day, week, however fine you need). This pre-crunched
data is then saved in a prdefined format. This resulting data is often
referred to a three dimensional dataset or what we call a data cube (1 of
the dimensions will usealy be time/dates).

There is a significant amount of technology and tools available from
Microsoft that will deal with this type of data and let you do some amazing
reporting on this data, and most is based around what is called pivot
tables.

This "add on" to sql server is called "analysis services".

You can do some reading here:

http://www.microsoft.com/sql/technologies/analysis/default.mspx

Signature

Albert D. Kallal    (Access MVP)
Edmonton, Alberta Canada
pleaseNOOSpamKallal@msn.com

Armen Stein - 17 May 2008 19:33 GMT
>I'm working a contract in the QA department of a well known multinational
>car company. Their QA survey data is surprisingly disorganized and I
[quoted text clipped - 26 lines]
>
>David Portwood

Hi David,

Lots of well-respected companies find themselves in this position.  We
work with them all the time.

You definitely need an industrial strength back-end database to do the
heavy lifting.  A powerful SQL Server machine can do a lot on the fly,
and as Albert pointed out, there are also ways to "pre-crunch" the
data so that common queries are already figured out.  This takes
planning and knowledge to put together.

Access can be a great front-end to all that SQL Server power.  Other
possibilities include SQL Server Reporting Services and ASP.NET web
applications.  It just depends on what you need.

The first step is to organize the structure of the raw data in a way
that will scale up to the volumes you need.  This structure can even
be done in Access as a model for small amounts of data, then
implemented in SQL Server later.  Then you can take the next steps
with the front-end applications and reporting.

If you don't have the experience to do this all yourself, you would be
wise to partner with a consultant to help.  A lot of people think that
they must choose either internal or external, but we've done
partnerships before where we helped a company's own internal resources
come up to speed, and then reduced our involvement.  This approach
might be a possibility for you.

Hope this helps,

Armen Stein
Microsoft Access MVP
www.JStreetTech.com
David Portwood - 20 May 2008 02:49 GMT
Thanks for the info, guys. My boss seemed unsurprised when I told him that
using Access as a back end for all that data would not be feasible after
all. I explained why, using what you both told me. I think my boss had
previously discussed the possiblities with the IS department and they told
him a similar story, so hopefully I didn't come across like too much of a
fool. There's no question, however, that this is something I definitely
should have foreseen.

> I'm working a contract in the QA department of a well known multinational
> car company. Their QA survey data is surprisingly disorganized and I
[quoted text clipped - 26 lines]
>
> David Portwood
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2008 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.